Contact surface, length determination, image processing method, system and x-ray machine for femur and patella

Abstract

The present disclosure relates to a femur and patella contact surface, length determination, image processing method, system and X-ray machine, and relates to the technical field of knee X-ray processing. The femur and patella contact surface determination method comprises: acquiring a femur region mask image and a patella region mask image corresponding to a to-be-processed knee X-ray image; determining an overlapping region corresponding to a femur region mask in the femur region mask image and a patella region mask in the patella region mask image; and configuring the overlapping region as a femur and patella contact surface in the to-be-processed knee X-ray image. The embodiment of the present disclosure can realize knee X-ray image processing.

Description

Technical Field

[0001] This disclosure relates to the field of knee joint X-ray processing technology, and in particular to a method, system, and X-ray camera for determining the contact surface and length of the femur and patella, as well as image processing. Background Technology

[0002] Determining the contact surface length between the femur and tibia is of significant clinical value in assessing joint stability and function, diagnosing joint diseases and injuries, guiding treatment planning, and predicting disease progression and prognosis.

[0003] The patellar surface of the femur and the articular surface of the patella match to form the patellofemoral joint. Accurately determining the extent and shape of the contact surfaces helps assess whether the patella's movement trajectory is normal during knee flexion and extension. For example, abnormal contact surfaces (such as a shallow trochlear groove or patellar tilt) may lead to patellar trajectory deviation, increasing the risk of patellofemoral joint instability and even causing patellar dislocation or subluxation. Observing the contact surfaces of the femur and patella can reveal trochlear dysplasia, patellar cartilage wear, osteoarthritis, and other conditions. For example, an increased trochlear groove angle (over 145°) suggests trochlear dysplasia, which may increase pressure on the lateral side of the patella, leading to patellofemoral pain syndrome. For patients with patellofemoral joint pain or mild instability, accurately understanding the contact surface and its length helps in developing personalized rehabilitation plans. For example, strengthening the vastus medialis muscle can improve patellar trajectory and reduce joint pressure.

[0004] The length of the patella and its relative position to the femoral condyle (e.g., patellar height index) are important indicators for assessing knee biomechanics. An excessively long or short patella can affect the leverage of the quadriceps, leading to abnormal knee extension strength and impacting the efficiency and stability of daily activities (such as walking and climbing stairs). Changes in patellar length (e.g., high or low patella) are associated with various knee joint diseases. A high patella may increase pressure on the patellofemoral joint, making it prone to patellar cartilage damage; a low patella may affect knee flexion and extension function, increasing the risk of patellar tendinitis. When considering surgical interventions (e.g., patellar rearrangement, trochleoplasty), accurate measurement of the contact surface and length of the femur and patella is crucial for surgical planning. Surgical goals typically include restoring normal joint anatomy, optimizing pressure distribution on the contact surface, and improving joint stability and function. Changes in the contact surface and length may be associated with the progression of knee joint diseases. For example, long-term patellar tracking abnormalities can lead to increased wear and tear on the patellofemoral joint cartilage, accelerating the development of osteoarthritis. By regularly monitoring these indicators, doctors can assess disease progression, adjust treatment strategies in a timely manner, and improve patient prognosis.

[0005] Compared to computed tomography (CT), magnetic resonance imaging (MRI), and other imaging modalities, X-rays are better suited for the initial assessment of degenerative osteoarthritis and pain from car accident injuries. As the most widely used basic imaging method in orthopedics, X-ray imaging has become the preferred imaging device for initial knee examinations (such as detecting fractures, dislocations, and other abnormalities) and assessing clinical conditions such as osteoarthritis and bone destruction due to its widespread availability, low cost, and rapid convenience. However, X-rays, CT, and MRI are not conflicting but complementary for knee imaging. In particular, because X-rays are an overlapping imaging modality, there are blind spots in the visualization of intra-articular structures, and they remain insufficient in detecting subtle or hidden fractures. Therefore, minor fractures or bone injuries can be detected from three-dimensional (3D) CT images of the knee. From 3D CT images of the knee, bone injuries and tumors around the knee joint can be observed from multiple angles. However, 3D CT images of the knee joint show lower diagnostic sensitivity for changes in the muscles and ligaments around the knee joint, especially when cartilage changes or bone hyperplasia have not yet occurred. Similar to CT, MR imaging is also multi-parameter, multi-planar, and multi-directional, with higher resolution for soft tissues than for bone. Clinical examination of cartilage, meniscus, or muscle and ligament injuries, synovitis, and joint effusion can be performed using 3D knee MR images. However, MRI is expensive, time-consuming, and the equipment is complex to operate. Furthermore, information on knee joint movement function cannot currently be obtained from MRI. Compared to dynamic knee X-rays, static knee X-ray images taken at a single moment lack information on knee joint movement, which is detrimental to assessing knee joint function.

[0006] Therefore, there is an urgent need to propose a technical solution for determining the contact surface and length of the femur and patella based on knee X-ray images and for image processing, in order to solve at least one of the technical problems, such as the inability to determine the contact surface and length of the femur and patella or the poor accuracy in determining the contact surface and length of the femur and patella in existing technologies, which hinder the assessment of joint stability and function, diagnosis of joint diseases and injuries, guidance of treatment plan formulation, and prediction of disease progression and prognosis. Summary of the Invention

[0007] This disclosure presents a technical solution for determining the contact surface and length of the femur and patella, an image processing method, a system, and a corresponding X-ray camera.

[0008] According to one aspect of this disclosure, a method for determining the contact surface between the femur and patella is provided, comprising: acquiring a femoral region mask image and a patellar region mask image corresponding to an X-ray image of the knee joint to be processed; determining an overlapping region corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; and configuring the overlapping region as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed.

[0009] Preferably, before obtaining the femoral region mask image and patellar region mask image corresponding to the knee X-ray image, the method includes: segmenting the femur and patella of the knee X-ray image to be processed using a knee X-ray imaging segmentation model to obtain the femoral region mask image and patellar region mask image corresponding to the knee X-ray image to be processed.

[0010] Preferably, constructing the knee joint X-ray image segmentation model includes: training a segmentation network using the set of multiple knee joint X-ray images or the set of dynamic knee joint X-ray images and their corresponding mask label images to construct the knee joint X-ray image segmentation model; wherein, the mask label images include: femoral mask labels and patellar mask labels.

[0011] Preferably, constructing the knee joint X-ray image segmentation model includes: training multiple segmentation networks using the set of multiple knee joint X-ray images or the set of dynamic knee joint X-ray images and their corresponding mask label images to obtain multiple knee joint X-ray image segmentation models; evaluating the multiple knee joint X-ray image segmentation models to determine the corresponding optimal knee joint X-ray image segmentation model; and segmenting the patella and femur and the patella in the dynamic knee joint X-ray image to be segmented based on the optimal knee joint X-ray image segmentation model.

[0012] Preferably, determining the corresponding optimal knee X-ray image segmentation model includes: determining multiple first comprehensive evaluation scales corresponding to multiple evaluation scales of multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models on a set knee X-ray image test set, wherein the values ​​of these scales are positively proportional or positively correlated with the performance of the segmentation models, and multiple second comprehensive evaluation scales whose values ​​are inversely proportional or negatively correlated with the performance of the segmentation models; determining, respectively, a first score vector corresponding to the multiple knee X-ray image segmentation models under the multiple first comprehensive evaluation scales and a second score vector corresponding to the multiple dynamic knee X-ray image segmentation models under the multiple second comprehensive evaluation scales; and evaluating the multiple knee X-ray image segmentation models based on the first score vector and the second score vector to determine the corresponding optimal knee X-ray image segmentation model.

[0013] Preferably, the training process for a set segmentation network or multiple set segmentation networks includes: acquiring the mask label image corresponding to the set dynamic knee X-ray image validation set and setting a maximum number of iterations; if the number of iterations of the set segmentation network is less than the maximum number of iterations, and the loss value between the set multiple knee X-ray images training set and its corresponding mask label image is less than a set loss value, and the first average intersection-union ratio (IUU) between the set multiple knee X-ray images validation set and its corresponding mask label image at the first set number of iterations is greater than the first average IUU set value, then the second average IUU ratio of the smallest mask segmentation region in the N mask segmentation regions corresponding to the mask label image and the smallest mask segmentation region in the set dynamic knee X-ray image validation set is greater than the second average IUU set value, then the set segmentation network is controlled to stop training.

[0014] Preferably, during the training of the set segmentation network or multiple set segmentation networks, the method further includes: if the number of iterations of the set segmentation network reaches the set maximum number of iterations, then controlling the set segmentation network to stop training; and / or, during the training of the set segmentation network or multiple set segmentation networks, the method further includes: if the number of iterations of the set segmentation network or multiple set segmentation networks is less than the set maximum number of iterations and the loss value between the set of multiple knee X-ray images training set and their corresponding mask label images is less than a set loss value and the set of multiple knee X-ray images validation set and their corresponding mask label images... If the first average intersection-union ratio (IUU) between the images is greater than the first average IUU set value at the first set number of iterations, and the second average IUU between the smallest area mask segmentation region in the N mask segmentation regions corresponding to the mask label image and the smallest area mask segmentation region in the training and validation sets of multiple knee X-ray images is greater than the second average IUU set value, then the third average IUU between the images and their corresponding mask label images in the training and validation sets of multiple knee X-ray images within the second set number of iterations is calculated; if the third average IUU is less than or equal to the set fluctuation value, then the set segmentation network is controlled to stop training.

[0015] Preferably, the method further includes: constructing a multi-segmentation region weight loss function based on the loss function weights of each of the N masked segmentation regions; and training the specified segmentation network using the specified training set of multiple knee X-ray images or the specified dynamic knee X-ray image training set and its corresponding mask label images, and the multi-segmentation region weight loss function.

[0016] Preferably, the step of constructing a multi-segmentation region weight loss function based on the loss function weights of each masked segmentation region in the training set of multiple knee X-ray images includes: obtaining the base loss function corresponding to each masked segmentation region; multiplying the loss function weights of each masked segmentation region by their respective base loss functions and summing the results to construct the multi-segmentation region weight loss function; wherein the base loss function corresponding to the loss function weights is configured as a cross-entropy loss function or a weighted cross-entropy loss function.

[0017] Preferably, determining the loss function weights for each masked segmentation region includes: determining the areas of N masked segmentation regions corresponding to the masked label images in the knee joint X-ray image training set; calculating N ratios between the area of ​​each of the N masked segmentation regions and the total area of ​​the masked segmentation regions corresponding to the N masked segmentation regions; and determining the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on N different products corresponding to any N-1 masked segmentation region areas among the N masked segmentation regions.

[0018] Preferably, the method further includes: if the training set of multiple knee X-ray images is configured as a dynamic knee X-ray image training set, then the similarity between multiple pre-labeled knee X-ray images and the multiple pre-labeled knee X-ray images in each example of the dynamic knee X-ray image is calculated; if the multiple image similarity is less than or equal to a preset image similarity, then the knee X-ray image corresponding to the preset image similarity is determined as the dynamic knee X-ray image to be labeled.

[0019] Preferably, the method further includes: if the initial preset labeled X-ray image corresponding to the plurality of preset labeled X-ray images is configured as the X-ray image corresponding to a non-initial time in each example of dynamic X-ray images, then the similarity between the initial preset labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the plurality of times before the initial preset labeled X-ray image is also calculated.

[0020] Preferably, the method further includes: if the last preset-labeled X-ray image corresponding to the plurality of preset-labeled X-ray images is configured as the X-ray image corresponding to a non-last moment in each example of dynamic X-ray images, then the similarity between the last preset-labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the moments after the last preset-labeled X-ray image is also calculated.

[0021] According to one aspect of this disclosure, a system for determining the contact surface length of the femur and patella is provided, comprising: determining the contact surface of the femur and patella in an X-ray image of the knee joint to be processed using the contact surface determination method described above; and determining the contact surface length of the femur and patella based on the contact surface.

[0022] Preferably, determining the contact surface length of the femur and patella based on the contact surface includes: calculating multiple non-zero coordinate distances between non-zero coordinates in the contact surface; and determining the maximum distance among the multiple non-zero coordinate distances as the contact surface length of the femur and patella.

[0023] Preferably, the step of calculating the distances between multiple non-zero coordinates in the contact surface includes: constructing a vector array using the non-zero coordinates in the contact surface; and calculating the distances between the currently traversed non-zero coordinate and other non-zero coordinates by traversing each non-zero coordinate in the vector array.

[0024] According to one aspect of this disclosure, a method for processing X-ray images of the knee joint is provided, comprising: the method for determining the contact surface of the femur and patella as described above; and / or, the method for determining the length of the contact surface of the femur and patella as described above; and, Extract the femoral mask boundary image corresponding to the femoral mask from the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, determine one or both of the first and second boundary lines among the boundary lines on both sides of the femur or femoral cortex; and / or, take the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, determine the first and second boundary lines among the boundary lines on both sides of the femur or femoral cortex; based on the first and second boundary lines, determine the femoral midline; and / or, using the keypoint detection model corresponding to the keypoint detection network, detect the third keypoint position coordinates corresponding to the highest point coordinates of the upper edge of the patella and the fourth keypoint position coordinates corresponding to the lowest point coordinates of the lower edge of the patella in the knee X-ray image to be processed, and determine the third and fourth keypoint position coordinates; based on the third and fourth keypoint position coordinates, determine the patellar length of the knee X-ray image to be processed.

[0025] According to one aspect of this disclosure, a knee joint X-ray imaging image processing system is provided, comprising: an acquisition unit for acquiring a femoral region mask image and a patellar region mask image corresponding to a knee joint X-ray image to be processed; a first determination unit for determining an overlapping region corresponding to the femoral region mask in the femoral region mask image and the patellar region mask image; a configuration unit for configuring the overlapping region as the contact surface between the femur and patella in the knee joint X-ray image to be processed; and / or The system includes: an acquisition unit for acquiring a femoral region mask image and a patellar region mask image corresponding to an X-ray image of the knee joint to be processed; a determination unit for determining the overlapping region corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; a configuration unit for configuring the overlapping region as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed; and a second determination unit for determining the length of the contact surface between the femur and patella based on the contact surface; and... An extraction unit is used to extract the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, a third determining unit is used to determine the first boundary line and the second boundary line among the boundary lines on both sides of the femur or femoral cortex; a fourth determining unit is used to determine the femoral midline based on the first boundary line and the second boundary line; and / or, using the key point detection model corresponding to the key point detection network, to detect the third key point position coordinates corresponding to the highest point coordinates of the upper edge of the patella and the fourth key point position coordinates corresponding to the lowest point coordinates of the lower edge of the patella in the knee X-ray image to be processed, and determine the third key point position coordinates and the fourth key point position coordinates; based on the third key point position coordinates and the fourth key point position coordinates, determine the patella length of the knee X-ray image to be processed.

[0026] According to one aspect of this disclosure, an X-ray camera is provided, including: the knee joint X-ray imaging image processing system as described above.

[0027] According to one aspect of this disclosure, an X-ray camera is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above-described method for determining the contact surface of the femur and patella; or, comprising: a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the above-described method for determining the contact surface of the femur and patella to generate the bit stream; or, comprising: a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the above-described method for determining the contact surface of the femur and patella.

[0028] According to one aspect of this disclosure, an X-ray machine is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the aforementioned method for determining the contact surface length of the femur and patella; or, comprising: a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the aforementioned method for determining the contact surface length of the femur and patella to generate the bit stream; or, comprising: a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the aforementioned method for determining the contact surface length of the femur and patella.

[0029] According to one aspect of this disclosure, an X-ray camera is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described knee joint X-ray image processing method; or, comprising: a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the above-described knee joint X-ray image processing method to generate the bit stream; or, comprising: a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the above-described knee joint X-ray image processing method.

[0030] In this disclosure, a technical solution is proposed for determining the contact surface and length of the femur and patella, an image processing method, a system, and a corresponding X-ray camera, to solve at least one of the technical problems in which the contact surface and length of the femur and patella cannot be determined or the accuracy of determining the contact surface and length of the femur and patella in the prior art is poor, which hinders the assessment of joint stability and function, diagnosis of joint diseases and injuries, guidance of treatment plan formulation, and prediction of disease progression and prognosis.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0032] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0034] Figure 1 A flowchart illustrating a method for determining the contact surface between the femur and patella according to an embodiment of the present disclosure is shown. Detailed Implementation

[0035] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0036] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0037] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0038] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0039] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further.

[0040] In addition, this disclosure also provides an image processing device or system, electronic device, computer-readable storage medium, and program corresponding to the contact surface and length determination of the femur and patella, and the image processing method. All of the above can be used to implement any of the image processing methods corresponding to the contact surface and length determination of the femur and patella provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section of the method and will not be repeated here.

[0041] Figure 1 A flowchart illustrating a method for determining the contact surface between the femur and patella according to an embodiment of this disclosure is shown. Figure 1As shown, the method for determining the contact surface between the femur and patella includes: Step S101: acquiring a femoral region mask image and a patellar region mask image corresponding to the X-ray image of the knee joint to be processed; Step S102: determining the overlapping area corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; Step S103: configuring the overlapping area as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed. This method aims to solve at least one of the technical problems currently facing the inability to determine the contact surface and length of the femur and patella, or the inaccuracy in existing technologies in determining the contact surface and length of the femur and patella, which hinders the assessment of joint stability and function, diagnosis of joint diseases and injuries, guidance of treatment plan formulation, and prediction of disease progression and prognosis.

[0042] Before obtaining the femoral region mask image and patellar region mask image corresponding to the knee X-ray image, the procedure includes: segmenting the femur and patella of the knee X-ray image to be processed using a knee X-ray imaging segmentation model to obtain the femoral region mask image and patellar region mask image corresponding to the knee X-ray image to be processed.

[0043] Constructing the knee joint X-ray image segmentation model includes: training a segmentation network using the set of multiple knee joint X-ray images or the set of dynamic knee joint X-ray images and their corresponding mask label images to construct the knee joint X-ray image segmentation model; wherein, the mask label images include: femoral mask label and patellar mask label.

[0044] Constructing the knee joint X-ray image segmentation model includes: training multiple segmentation networks using the specified training set of multiple knee joint X-ray images or the specified training set of dynamic knee joint X-ray images and their corresponding mask label images to obtain multiple knee joint X-ray image segmentation models; evaluating the multiple knee joint X-ray image segmentation models to determine the corresponding optimal knee joint X-ray image segmentation model; and segmenting the patella and femur and the patella based on the optimal knee joint X-ray image segmentation model.

[0045] The step of determining the optimal knee X-ray image segmentation model includes: determining multiple first comprehensive evaluation scales corresponding to multiple evaluation scales of multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models on a set of knee X-ray images, where the values ​​of these scales are positively proportional or positively correlated with the performance of the segmentation models, and multiple second comprehensive evaluation scales where the values ​​of these scales are inversely proportional or negatively correlated with the performance of the segmentation models; determining first score vectors corresponding to the multiple knee X-ray image segmentation models under the multiple first comprehensive evaluation scales and second score vectors corresponding to the multiple dynamic knee X-ray image segmentation models under the multiple second comprehensive evaluation scales; and evaluating the multiple knee X-ray image segmentation models based on the first score vectors and the second score vectors to determine the optimal knee X-ray image segmentation model.

[0046] During the training of a set segmentation network or multiple set segmentation networks, the process includes: obtaining the mask label image corresponding to the set dynamic knee X-ray image validation set and setting a maximum number of iterations; if the number of iterations of the set segmentation network is less than the maximum number of iterations, and the loss value between the set multiple knee X-ray images training set and its corresponding mask label image is less than a set loss value, and the first average intersection-union ratio (IUU) between the set multiple knee X-ray images validation set and its corresponding mask label image at the first set number of iterations is greater than the first average IUU set value, then the second average IUU ratio of the smallest mask segmentation region in the N mask segmentation regions corresponding to the mask label image and the smallest mask segmentation region in the set dynamic knee X-ray image validation set is greater than the second average IUU set value, then the set segmentation network is controlled to stop training.

[0047] The training process for the specified segmentation network or multiple specified segmentation networks further includes: if the number of iterations of the specified segmentation network reaches the specified maximum number of iterations, then controlling the specified segmentation network to stop training; and / or, the training process for the specified segmentation network or multiple specified segmentation networks further includes: if the number of iterations of the specified segmentation network or multiple specified segmentation networks is less than the specified maximum number of iterations and the loss value between the specified multiple knee X-ray images training set and its corresponding mask label image is less than a specified loss value and the loss value between the specified multiple knee X-ray images validation set and its corresponding mask label image... If the first average intersection-union ratio (IUU) corresponding to the first set number of iterations is greater than the first average IUU set value, and the second average IUU corresponding to the smallest area mask segmentation region among the N mask segmentation regions corresponding to the mask label image and the smallest area mask segmentation region in the training and validation sets of multiple knee X-ray images is greater than the second average IUU set value, then the third average IUU between the training and validation sets of multiple knee X-ray images and their corresponding mask label images is calculated within the second set number of iterations; if the third average IUU is less than or equal to the set fluctuation value, then the set segmentation network is controlled to stop training.

[0048] It also includes: constructing a multi-segmentation region weight loss function based on the loss function weights of each of the N masked segmentation regions; and training the set segmentation network using the set of multiple knee X-ray images or the set of dynamic knee X-ray images and their corresponding masked label images, and the multi-segmentation region weight loss function.

[0049] The step of constructing a multi-segmentation region weighted loss function based on the loss function weights of each masked segmentation region in the training set of multiple knee X-ray images includes: obtaining the base loss function corresponding to each masked segmentation region; multiplying the loss function weights of each masked segmentation region by their respective base loss functions and summing the results to construct the multi-segmentation region weighted loss function; wherein the base loss function corresponding to the loss function weights is configured as a cross-entropy loss function or a weighted cross-entropy loss function.

[0050] Determining the loss function weights for each masked segmentation region includes: determining the areas of N masked segmentation regions corresponding to the masked label images in the knee joint X-ray image training set; calculating N ratios between the area of ​​each of the N masked segmentation regions and the total area of ​​the masked segmentation regions corresponding to the N masked segmentation regions; and determining the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on N different products corresponding to any N-1 masked segmentation region areas among the N masked segmentation regions.

[0051] It also includes: if the training set of multiple knee X-ray images is configured as a dynamic knee X-ray image training set, then calculate the similarity between multiple pre-labeled knee X-ray images and the multiple pre-labeled knee X-ray images in each example of the dynamic knee X-ray image; if the multiple image similarity is less than or equal to the preset image similarity, then determine the knee X-ray image corresponding to the preset image similarity as the dynamic knee X-ray image to be labeled.

[0052] It also includes: if the initial preset labeled X-ray image corresponding to the plurality of preset labeled X-ray images is configured as the X-ray image corresponding to a non-initial time in each example of dynamic X-ray images, then the similarity between the initial preset labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the plurality of times before the initial preset labeled X-ray image is also calculated.

[0053] It also includes: if the last preset-labeled X-ray image corresponding to the plurality of preset-labeled X-ray images is configured as the X-ray image corresponding to a non-last moment in each example of dynamic X-ray images, then the similarity between the last preset-labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the times after the last preset-labeled X-ray image is also calculated.

[0054] This disclosure proposes a dynamic knee X-ray image segmentation method, comprising: constructing a multi-segmentation region weight loss function based on a set of multiple knee X-ray images or a set of dynamic knee X-ray images corresponding to a mask label image; wherein the mask label image includes one or more of the following: patellar mask label, femoral mask label, tibial mask label, and patellar tendon mask label; and using the set of multiple knee X-ray images or the set of dynamic knee X-ray images and their corresponding mask label images, and the multi-segmentation region weight loss function, to perform segmentation on multiple defined segmentation networks. Training is performed to obtain multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image (multi-time dynamic knee X-ray image) segmentation models. These multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models are evaluated to determine the corresponding optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model. Based on the optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model, the patella, femur, tibia, and patellar tendon are segmented from the dynamic knee joint X-ray image to be segmented. This addresses the technical problem that existing technologies cannot accurately and automatically segment one or more of the patella, femur, and tibia, and the patellar tendon.

[0055] In this embodiment, a multi-segmentation region weight loss function is constructed based on a training set of multiple knee X-ray images or a training set of dynamic knee X-ray images corresponding to masked label images. The masked label images include one or more of the following: patellar mask label, femoral mask label, tibial mask label, and patellar tendon mask label. This addresses the technical problem of unbalanced area ratios among the N masked segmentation regions corresponding to the masked label images, resulting in insufficient training of smaller masked segmentation regions, thereby improving the segmentation performance of smaller masked segmentation regions.

[0056] For example, in a training set of multi-time dynamic knee X-ray images (multi-time dynamic knee X-ray imaging images), there is a technical problem that the area ratios of the patellar, femoral, tibial, and patellar tendon masked segments corresponding to the N masked segmentation areas of the masked label images are unbalanced, resulting in the smaller patellar tendon masked segmentation area not being sufficiently trained.

[0057] In embodiments of this disclosure, the step of constructing a multi-segmentation region weight loss function based on a set of multiple knee X-ray images or a set of dynamic knee X-ray images corresponding to a masked label image includes: determining the areas of N masked segmentation regions corresponding to the masked label image in the dynamic knee X-ray image training set; wherein N is greater than 1 or greater than or equal to 2, and N is a positive integer; calculating N ratios between the area of ​​each of the N masked segmentation regions and the total area of ​​the N masked segmentation regions; and determining the loss function weights of each masked segmentation region corresponding to the set segmentation network based on N different products corresponding to any N-1 masked segmentation regions among the N masked segmentation regions.

[0058] In embodiments of this disclosure, determining the loss function weight of each mask segmentation region corresponding to the defined segmentation network based on N different products corresponding to any N-1 mask segmentation region areas among the N mask segmentation region areas includes: calculating N products corresponding to any N-1 mask segmentation region areas among the N mask segmentation region areas; and determining the loss function weight of each mask segmentation region corresponding to the defined segmentation network based on the N products and the sum of the products corresponding to the N products.

[0059] In the embodiments of this disclosure, in determining the loss function weight of each mask segmentation region corresponding to the set segmentation network based on N different products corresponding to any N-1 mask segmentation region areas among the N mask segmentation region areas, determining the loss function weight of each mask segmentation region corresponding to the set segmentation network based on the N products and the sum of the products corresponding to the N products includes: extracting the product of the mask segmentation region areas that do not contain the weight to be determined from the N different products; calculating the ratio of the product of the mask segmentation region areas that contain the weight to be determined to the sum of the products corresponding to the N different products, and determining the loss function weight of each mask segmentation region corresponding to the set segmentation network.

[0060] For example, the areas of the N masked segmentation regions are configured as A, B, C, and D; calculate N ratios a (a=A / (A+B+C+D)), b (b=B / (A+B+C+D)), c (c=C / (A+B+C+D)), and d (d=D / (A+B+C+D)) for each of the N masked segmentation regions A, B, C, and D, respectively, to the total area of ​​the N masked segmentation regions (A+B+C+D); calculate N different products a*b*c, a*c*d, a*b*d, and b*c*d corresponding to any N-1 masked segmentation region areas among the N masked segmentation regions. Extract the products b*c*d, a*c*d, a*b*d, and a*b*c of the areas of the masked regions A, B, C, and D that do not contain the weight to be determined from the N different products a*b*c, a*c*d, a*b*d, and b*c*d, respectively. Then, calculate the sum of the products b*c*d, a*c*d, a*b*d, and a*b*c of the areas of the masked regions whose weights are to be determined, and the sum of the products of the N different products (a) and b*c*d. The ratio of *b*c+a*c*d+a*b*d+b*c*d is used to determine the loss function weights b*c*d / (a*b*c+a*c*d+a*b*d+b*c*d), a*c*d / (a*b*c+a*c*d+a*b*d+b*c*d), a*b*d / (a*b*c+a*c*d+a*b*d+b*c*d), and a*b*c / (a*b*c+a*c*d+a*b*d+b*c*d) for each mask segmentation region corresponding to the set segmentation network.

[0061] In embodiments of this disclosure, determining the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on N different products corresponding to any N-1 masked segmentation region areas among the N masked segmentation region areas includes: multiplying the N masked segmentation region areas by a plurality of corresponding weight variables to be optimized to obtain N masked segmentation region weight areas; setting the N masked segmentation region weight areas equal to a set value to obtain the N weight variables to be optimized based on the set value and the N masked segmentation region areas; and determining the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on the set value and the N weight variables to be optimized based on the N masked segmentation region areas.

[0062] In the embodiments of this disclosure, when determining the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on N different products corresponding to any N-1 masked segmentation region areas among the N masked segmentation region areas, the determination of the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on the defined value and the N masked segmentation region areas includes: calculating the sum of the weight variables to be optimized corresponding to the defined value and the N masked segmentation region areas to obtain the weight variables to be optimized and their expressions; and calculating the ratio between each of the N weight variables to be optimized and their expressions to determine the loss function weights for each masked segmentation region corresponding to the defined segmentation network.

[0063] For example, the areas of the N mask segmentation regions are configured as A, B, C, and D; the areas of the N mask segmentation regions are multiplied by the corresponding multiple weight variables to be optimized ε1, ε2, ε3, and ε4 respectively to obtain the weight areas of the N mask segmentation regions ε1*A, ε2*B, ε3*C, and ε4*D; the weight areas of the N mask segmentation regions ε1*A, ε2*B, ε3*C, and ε4*D are set to equal a predetermined value ε, resulting in the N weight variables to be optimized ε1=ε / A, ε2=ε / B, ε3=ε / C, and ε4=ε / D, represented by the predetermined value and the areas of the N mask segmentation regions; Calculate the sum of the N weight variables to be optimized, represented by the set value and the areas of the N masked segmentation regions, to obtain the weight variables to be optimized and the expression (ε1+ε2+ε3+ε4=ε / A+ε / B+ε / C+ε / D); calculate the ratio ε1 / (ε1+ε2+ε3+ε4) between each of the N weight variables to be optimized represented by the set value and the areas of the N masked segmentation regions and the weight variables to be optimized and the expression. )=(ε / A) / (ε / A+ε / B+ε / C+ε / D), ε2 / (ε1+ε2+ε3+ε4)=(ε / B) / (ε / A+ε / B+ε / C+ε / D), ε3 / (ε1+ε2+ε3+ε4)=(ε / C) / (ε / A+ε / B+ε / C+ε / D), ε4 / (ε1+ε2+ε3+ε4)=(ε / D) / (ε / A+ε / B+ε / C+ε / D), determine the loss function weight of each mask segmentation area corresponding to the set segmentation network.

[0064] In this embodiment of the disclosure, multiple segmentation networks are trained using the set of multiple knee X-ray images or the set of dynamic knee X-ray images and their corresponding mask label images, and the multi-segmentation region weight loss function, to obtain multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image (multi-time dynamic knee X-ray images) segmentation models.

[0065] In embodiments of this disclosure, the plurality of segmentation networks are configured as one or more of FCN, UNet, UPerNet, SegFormer, PSPNet, DeepLabV3, or a combination thereof.

[0066] In the embodiments of this disclosure, the training set of the set dynamic knee X-ray images and their corresponding mask label images, and the weight loss function of the multi-segmentation region are used to train one or more of the following, or a combination of them, of multiple set segmentation networks: FCN, UNet, UPerNet, SegFormer, PSPNet, and DeepLabV3, to obtain multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image (multi-time dynamic knee X-ray image) segmentation models.

[0067] In this embodiment of the disclosure, the plurality of knee joint X-ray image segmentation models or the plurality of dynamic knee joint X-ray image segmentation models are evaluated to determine the corresponding optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model.

[0068] In the embodiments of this disclosure, the evaluation of the plurality of dynamic knee X-ray image segmentation models to determine the corresponding optimal knee X-ray image segmentation model or optimal dynamic knee X-ray image segmentation model includes: determining the plurality of knee X-ray image segmentation models corresponding to the plurality of set segmentation networks under a set dynamic knee X-ray training set and a multi-segmentation region weight loss function (set loss function), or the plurality of dynamic knee X-ray image segmentation models on a set knee X-ray image test set, or the plurality of evaluation scales corresponding to the plurality of dynamic knee X-ray image test sets, whose values ​​are directly proportional or positively correlated with the performance of the segmentation model (the larger the value, the better the performance of the segmentation model), and whose values ​​are directly proportional to or positively correlated with the performance of the segmentation model. Multiple second comprehensive evaluation scales with inverse or negative correlation (the smaller the value, the better the segmentation model performance) are used. First score vectors are determined for the multiple knee X-ray image segmentation models or the multiple dynamic knee X-ray image segmentation models under the multiple first comprehensive evaluation scales, and second score vectors are determined for the multiple dynamic knee X-ray image segmentation models under the multiple second comprehensive evaluation scales. Based on the first score vectors and the second score vectors, the multiple knee X-ray image segmentation models or the multiple dynamic knee X-ray image segmentation models are evaluated, and the corresponding optimal knee X-ray image segmentation model or optimal dynamic knee X-ray image segmentation model is determined. To address the technical problem that improvements to multiple knee X-ray image segmentation models or dynamic knee X-ray image segmentation models often fail to achieve across all evaluation scales, thus hindering the selection of appropriate models, the aim is to improve the performance of multiple knee X-ray image segmentation models or dynamic knee X-ray image segmentation models.

[0069] In this embodiment of the disclosure, the plurality of evaluation metrics are configured as any of the following: IoU (Intersection over Union), Dice, precision, recall, Hausdorff distance (HD) or median 95th Hausdorff distance (HD95), and average symmetric surface distance (ASSD); and / or, the plurality of first comprehensive evaluation metrics are configured as any one or more of IoU, Dice, precision, and recall; and / or, the plurality of second comprehensive evaluation metrics are any one or more of Hausdorff distance or median 95th Hausdorff distance and average symmetric surface distance.

[0070] In this embodiment of the disclosure, determining multiple knee X-ray image segmentation models corresponding to multiple set segmentation networks under a set of multiple knee X-ray images or a set of dynamic knee X-ray images and a set loss function, or multiple dynamic knee X-ray image segmentation models on a set of dynamic knee X-ray images, includes: calculating multiple knee X-ray image segmentation models corresponding to multiple set segmentation networks under a set of multiple knee X-ray images training set or a set of dynamic knee X-ray images test set, where the values ​​of these multiple evaluation metrics are positively proportional or positively correlated with the performance of the segmentation models, and multiple second comprehensive evaluation metrics where the values ​​of these metrics are inversely proportional or negatively correlated with the performance of the segmentation models. The dynamic knee X-ray image segmentation model uses multiple evaluation scales corresponding to a set of dynamic knee X-ray images. It then determines a first set of evaluation scales whose values ​​are positively proportional or correlated with the segmentation model performance, and a second set of evaluation scales whose values ​​are inversely proportional or negatively correlated with the segmentation model performance. Based on the first set of evaluation scales and the second set of evaluation scales, it determines multiple first comprehensive evaluation scales and multiple second comprehensive evaluation scales.

[0071] In this embodiment of the disclosure, determining multiple first comprehensive evaluation scales and multiple second comprehensive evaluation scales based on the first set of evaluation scales and the second set of evaluation scales respectively includes: summing the first set of evaluation scales corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models on a set of dynamic knee X-ray image test sets to determine multiple first comprehensive evaluation scales corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models; and summing the second set of evaluation scales corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models on a set of dynamic knee X-ray image test sets to determine multiple second comprehensive evaluation scales corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models.

[0072] In this embodiment of the disclosure, the step of determining multiple first comprehensive evaluation scales and multiple second comprehensive evaluation scales based on the first set of evaluation scales and the second set of evaluation scales respectively further includes: determining a first number corresponding to the evaluation scales in the first set of evaluation scales and a second number corresponding to the evaluation scales in the second set of evaluation scales respectively; averaging the multiple first comprehensive evaluation scales corresponding to the multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models using the first number respectively to obtain the final multiple first comprehensive evaluation scales; and averaging the multiple second comprehensive evaluation scales corresponding to the multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models using the second number respectively to obtain the final multiple second comprehensive evaluation scales.

[0073] In this embodiment of the disclosure, the step of determining the first score vector corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models under multiple first comprehensive evaluation scales and the second score vector corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models under multiple second comprehensive evaluation scales includes: sorting the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models according to the set order of the multiple first comprehensive evaluation scales (from largest to smallest or from smallest to largest) to determine the first score vector corresponding to the multiple dynamic knee X-ray image segmentation models under the multiple first comprehensive evaluation scales; and sorting the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models according to the set order of the multiple second comprehensive evaluation scales (from largest to smallest or from smallest to largest) to determine the second score vector corresponding to the multiple dynamic knee X-ray image segmentation models under the multiple second comprehensive evaluation scales.

[0074] In this embodiment of the disclosure, the step of determining the first score vector corresponding to the multiple dynamic knee X-ray image segmentation models under multiple first comprehensive evaluation scales and the second score vector corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models under multiple second comprehensive evaluation scales further includes: determining the maximum score value among the first score vector and the second score vector; determining the score corresponding to the knee X-ray image segmentation model or multiple dynamic knee X-ray image segmentation model with the largest first comprehensive evaluation scale among the multiple first comprehensive evaluation scales as the maximum score value; and sorting the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models according to a predetermined order (from largest to smallest or from smallest to largest) corresponding to the multiple first comprehensive evaluation scales, based on the maximum score value and a predetermined score difference between any two adjacent scores in the first score vector. The image segmentation model is scored and assigned a first score vector to determine the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models under multiple first comprehensive evaluation scales; the score corresponding to the knee X-ray image segmentation model or multiple dynamic knee X-ray image segmentation model with the smallest second comprehensive evaluation scale among the multiple second comprehensive evaluation scales is determined as the maximum score value; based on the maximum score value and the set score difference between any two adjacent scores in the second score vector, the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models, after being sorted according to the set order (from largest to smallest or from smallest to largest) corresponding to the multiple second comprehensive evaluation scales, are scored and assigned a second score vector to determine the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models under multiple second comprehensive evaluation scales.

[0075] In this embodiment of the disclosure, determining the maximum score value among the first score vector and the second score vector includes: determining the maximum score value among the first score vector and the second score vector based on the number of the plurality of knee joint X-ray image segmentation models or the plurality of dynamic knee joint X-ray image segmentation models.

[0076] For example, the maximum score value in the first and second score vectors is configured to be 50; or, the number of the multiple knee joint X-ray image segmentation models or the multiple dynamic knee joint X-ray image segmentation models is configured to be 50, then the maximum score value in the first and second score vectors is configured to be 50. In this case, the set score difference between any two adjacent scores in the first and second score vectors is configured to be 1, and the determined first and second score vectors are configured as [50, 49, 48, …, 3, 2, 1] or [1, 2, 3, …, 48, 49, 50], respectively.

[0077] In this embodiment of the disclosure, the evaluation of the plurality of knee joint X-ray image segmentation models or the plurality of dynamic knee joint X-ray image segmentation models based on the first scoring vector and the second scoring vector includes: constructing a correlation relationship between the first scoring vector and the second scoring vector according to the plurality of knee joint X-ray image segmentation models or the plurality of dynamic knee joint X-ray image segmentation models; calculating the sum of the first correlation score and the second correlation score in the first scoring vector and the second scoring vector of the plurality of knee joint X-ray image segmentation models or the plurality of dynamic knee joint X-ray image segmentation models under the correlation relationship, corresponding to a plurality of joint scores; and evaluating the plurality of knee joint X-ray image segmentation models or the plurality of dynamic knee joint X-ray image segmentation models based on the plurality of joint scores.

[0078] In this embodiment of the disclosure, the step of constructing the association between the first scoring vector and the second scoring vector based on the plurality of knee X-ray image segmentation models or the plurality of dynamic knee X-ray image segmentation models includes: extracting the first score and the second score corresponding to the same knee X-ray image segmentation model or the same dynamic knee X-ray image segmentation model from the first scoring vector and the second scoring vector, respectively, to construct the association between the first scoring vector and the second scoring vector; or, determining the association labels of the first score in the first scoring vector and the second score in the second scoring vector corresponding to the same dynamic knee X-ray image segmentation model in the plurality of dynamic knee X-ray image segmentation models, respectively; and constructing the association between the first scoring vector and the second scoring vector based on the association labels.

[0079] In this embodiment of the disclosure, the evaluation of the multiple knee joint X-ray image segmentation models or the multiple dynamic knee joint X-ray image segmentation models based on the multiple joint scores includes: sorting the multiple joint scores to obtain a sorted joint score vector; and evaluating the multiple dynamic knee joint X-ray image segmentation models based on the sorted joint score vector; wherein the joint score in the sorted joint score vector is proportional to the performance of the dynamic knee joint X-ray image segmentation model.

[0080] In this embodiment of the disclosure, determining multiple dynamic knee X-ray image segmentation models corresponding to multiple defined segmentation networks includes: determining target segmentation regions corresponding to the multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models; if the number of target segmentation regions is greater than 1, determining the areas of multiple masked segmentation regions corresponding to the masked label images in the training set of dynamic knee X-ray images corresponding to the multiple defined segmentation networks; if the difference or ratio between the largest and smallest masked segmentation region areas among the multiple masked segmentation region areas is greater than a set difference or a set ratio, optimizing the defined loss function corresponding to the multiple defined segmentation networks to obtain an optimized loss function; and training the multiple defined segmentation networks based on the dynamic knee X-ray image training set and the optimized loss function to determine multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models corresponding to the multiple defined segmentation networks.

[0081] In this embodiment, optimizing the set loss function corresponding to the plurality of set segmentation networks to obtain an optimized loss function includes: determining the areas of N masked segmentation regions corresponding to the masked label images in a set of multiple knee X-ray images or a set of dynamic knee X-ray images training sets corresponding to the set segmentation networks; wherein N is greater than 1 or greater than or equal to 2, and N is a positive integer; calculating N ratios of each masked segmentation region area to the total area of ​​the N masked segmentation regions; and determining the loss function weights of each masked segmentation region corresponding to the set segmentation network based on N different products corresponding to any N-1 masked segmentation region areas, thereby obtaining the optimized loss function. This addresses the technical problem that the unbalanced proportion of the N masked segmentation region areas corresponding to the masked label images prevents smaller masked segmentation regions from being adequately trained, thereby improving the segmentation performance of smaller masked segmentation regions.

[0082] In this embodiment of the disclosure, training multiple segmentation networks using the set of multiple knee X-ray images or the set of dynamic knee X-ray images and their corresponding masked images, and the multi-segmentation region weight loss function, includes: obtaining a set of dynamic knee X-ray images and the corresponding masked images of the set of dynamic knee X-ray images, and setting a maximum number of iterations; if the number of iterations of the segmentation network is less than the set maximum number of iterations and the set of multiple knee X-ray images or the set of dynamic knee X-ray images... If the loss value between the training set of X-ray images and their corresponding masked images is less than a set loss value, and the first average intersection-union ratio (IU) between the set dynamic knee X-ray image validation set and its corresponding masked images at a first set number of iterations is greater than a set first average IU, then the second average IU between the smallest masked segmentation region among the N masked segmentation regions corresponding to the masked images and the smallest masked segmentation region (patellar tendon) in the set dynamic knee X-ray image validation set is greater than a set second average IU, then the set segmentation network is controlled to stop training. This addresses at least one technical problem in multi-target region segmentation models, such as overfitting, wasted computational resources and time, difficulty in capturing the optimal state of the multi-target region segmentation model, and poor robustness of the multi-target region segmentation model.

[0083] In this embodiment of the disclosure, the Intersection over Union (IoU) ratio is the ratio of the intersection area to the union area of ​​two regions, and is used to measure the similarity between the predicted segmentation result and the actual segmentation result.

[0084] In this embodiment of the disclosure, it further includes: if the number of iterations of the set segmentation network reaches the set maximum number of iterations, then controlling the set segmentation network to stop training.

[0085] In this embodiment of the disclosure, the method further includes: if the number of iterations of the defined segmentation network is less than a defined maximum number of iterations, and the loss value between the defined training set of multiple knee X-ray images or the defined dynamic knee X-ray image training set and their corresponding mask label images is less than a defined loss value, and the first average intersection-union ratio (IUU) between the defined dynamic knee X-ray image validation set and their corresponding mask label images at the first defined number of iterations is greater than a defined first average IUU value, and the second average IUU between the smallest mask segmentation region in the N mask segmentation regions corresponding to the mask label image and the smallest mask segmentation region in the defined dynamic knee X-ray image validation set is greater than a defined second average IUU value, then a third average IUUU between the defined dynamic knee X-ray image validation set and their corresponding mask label images is calculated within the second defined number of iterations; if the third average IUUU is less than or equal to a defined fluctuation value, then the defined segmentation network is controlled to stop training.

[0086] In this embodiment of the disclosure, the method further includes: constructing a multi-segmentation region weight loss function based on the loss function weights of each of the N masked segmentation regions; and training the set segmentation network using the set of multiple knee X-ray images or the set of dynamic knee X-ray images and their corresponding masked label images, and the multi-segmentation region weight loss function.

[0087] In this embodiment of the disclosure, the step of constructing a multi-segmentation region weighted loss function based on the loss function weights of each masked segmentation region includes: obtaining the base loss function corresponding to each masked segmentation region; multiplying the loss function weights of each masked segmentation region by their respective base loss functions and summing the results to construct the multi-segmentation region weighted loss function; wherein the base loss function corresponding to the loss function weights is configured as a cross-entropy loss function or a weighted cross-entropy loss function.

[0088] In this embodiment, the method further includes: summing the boundary loss functions corresponding to each masked segmentation region or summing the corresponding multi-segmentation region boundary loss function after multiplying the loss function weights of each masked segmentation region by their respective boundary loss functions; summing the DICE loss functions corresponding to each masked segmentation region or summing the corresponding multi-segmentation region DICE loss function after multiplying the loss function weights of each masked segmentation region by their respective DICE loss functions; constructing a multivariate loss function based on one or two of the segmentation region weight loss function, multi-segmentation region boundary loss function, or multi-segmentation region DICE loss function; and training the specified segmentation network using the specified training set of multiple knee X-ray images or the specified dynamic knee X-ray image training set and its corresponding mask label images, and the multivariate loss function. By using the multivariate loss function corresponding to the multivariate loss function and one or two of the multivariate loss functions (multi-segmentation region weight loss function and boundary loss function or multi-segmentation region DICE loss function), the parameter training direction of the specified segmentation network is guided, thereby solving the technical problem that a univariate loss function cannot comprehensively train the specified segmentation network.

[0089] For example, firstly, one of the basic criteria for stopping network training is that the loss value between the knee multi-object segmentation images (setting a training set of multiple knee X-ray images or setting a training set of dynamic knee X-ray images) and the ground truth (GT) (mask label / mask label image) in its training set is less than 0.2 (loss setting). Secondly, the overall segmentation performance of the model in four object regions (segmentation regions corresponding to the patella, femur, tibia, and patellar tendon) is evaluated every 50 iterations (first set number of iterations) using a validation set and the knee multi-object segmentation model. Based on the loss value between the knee multi-object segmentation images and the GT in its training set being less than 0.2, a first average intersection-over-union (IoU) ratio greater than 0.8 (first average IoU setting) between the knee multi-object segmentation images and the GTs in its validation set is set as the second basic criterion for stopping network training. Thirdly, due to the small area of ​​the patellar tendon (the smallest mask segmentation region), it is difficult to achieve good performance. To provide more comprehensive training for the patellar tendon, a third criterion for stopping training is set: an average IoU greater than 0.85 (the second average intersection-union ratio setting) based on the patellar tendon object segmentation images and their ground truth (GT). Finally, after meeting these three criteria, network training can be stopped if the total average IoU between the four object segmentation regions and their GTs in the validation set remains unchanged for 10 consecutive iterations (the second set number of iterations). This means the relative fluctuation of the total average IoU between the four object segmentation regions and their GTs in the ten validation sets is less than or equal to the set fluctuation value of 0.001. If these three criteria are not met, network training stops when the maximum number of iterations is reached. After network training is complete, the segmentation model that performs best on the validation set is selected for testing on the test set.

[0090] In this embodiment of the disclosure, the method further includes: dividing the learning rate decay of the optimizer corresponding to the specified segmentation network into an initial learning rate decay phase, a stable learning rate decay period, and a fine-tuning learning rate decay phase; wherein, the initial learning rate decay phase uses warm-up steps to ensure rapid gradient descent in the initial stage of training the specified segmentation network while maintaining the stability of the specified segmentation network; the stable learning rate decay period uses polynomial decay to maintain a relatively stable learning rate; and the fine-tuning learning rate decay phase uses cosine learning rate decay.

[0091] For example, in network training, the learning rate is a key hyperparameter determining model convergence. Dynamic learning rates, by adaptively adjusting the learning rate during training, can achieve rapid convergence early on and fine-tune later, resulting in better model performance. In this study, the AdamW optimizer or a modified AdamW optimizer was used with an initial learning rate of 1e-4. The AdamW optimizer's learning rate decay includes an initial learning rate decay phase, a stable learning rate decay phase, and a fine-tuning learning rate decay phase. The initial learning rate decay phase, based on the initial learning rate, uses warmup steps, which provides rapid gradient descent in the initial stages of network training while maintaining stability. The stable learning rate decay phase uses polynomial decay, maintaining a relatively stable learning rate during long-term network training. The fine-tuning learning rate decay phase uses cosine learning rate decay, applied at the end of training.

[0092] In this embodiment of the disclosure, the multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models corresponding to the multiple set segmentation networks are configured as multi-target segmentation regions that are consistent with the N mask segmentation regions corresponding to the mask label image.

[0093] In this embodiment of the disclosure, the multi-target segmentation region that corresponds to the N mask segmentation regions of the mask label image is configured as two or more of the following: patella, femur, tibia and patellar tendon.

[0094] In this embodiment of the disclosure, based on the optimal knee X-ray image segmentation model or the optimal dynamic knee X-ray image segmentation model, the dynamic knee X-ray image to be segmented or processed is segmented into one or more of the patella, femur, tibia and patellar tendon.

[0095] In this embodiment of the disclosure, the method further includes: before training multiple segmentation networks using a set of multiple knee X-ray images or a set of dynamic knee X-ray images and their corresponding mask labels to obtain multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models, the method includes: extracting each dynamic knee X-ray image from the preset dynamic knee X-ray image training set; wherein, each dynamic knee X-ray image includes: multiple knee X-ray images at multiple times corresponding to the same subject; calculating the similarity between multiple preset labeled knee X-ray images and multiple knee X-ray images corresponding to multiple knee X-ray images in each dynamic knee X-ray image; if the multiple image similarity is less than or equal to a preset image similarity, then the knee X-ray image corresponding to the preset image similarity is determined as a dynamic knee X-ray image to be labeled. To address the challenges of X-ray images with similarity levels below a preset threshold hindering the training of segmentation networks and preventing the acquisition of highly generalized segmentation models, the following measures are proposed: Firstly, determining the X-ray images to be labeled in the dynamic X-ray image training set according to preset time intervals introduces a high degree of subjectivity. Secondly, labeling all images in the dynamic X-ray image training set presents not only the technical problem of X-ray images with similarity levels below a preset threshold hindering the training of segmentation networks and preventing the acquisition of highly generalized segmentation models, but also presents at least one of the technical challenges inherent in the heavy physician annotation task. These measures aim to improve the generalization of the segmentation model.

[0096] In this embodiment of the disclosure, the image similarity can be configured as the mean square error corresponding to the average of the squared differences of the corresponding pixel values ​​of two images, extracting key points of the image and their descriptors, and calculating one or more of the similarity based on features, hash-based similarity, and statistical similarity (histogram similarity or structural similarity index) by matching descriptors.

[0097] In this embodiment of the disclosure, the first (first or first) preset labeled X-ray image among the plurality of preset labeled X-ray images is configured as the X-ray image corresponding to the initial moment in each dynamic X-ray image; and / or, the last (last) preset labeled X-ray image among the plurality of preset labeled X-ray images is configured as the X-ray image corresponding to the last moment in each dynamic X-ray image.

[0098] In this embodiment of the disclosure, it further includes: if the initial preset labeled X-ray image corresponding to the plurality of preset labeled X-ray images is configured as the X-ray image corresponding to a non-initial time in each example of dynamic X-ray images, then the similarity between the initial preset labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the plurality of times before the initial preset labeled X-ray image is also calculated.

[0099] In this embodiment of the disclosure, it further includes: if the last preset-labeled X-ray image corresponding to the plurality of preset-labeled X-ray images is configured as the X-ray image corresponding to a non-last moment in each example of dynamic X-ray images, then the similarity between the last preset-labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the times after the last preset-labeled X-ray image is also calculated.

[0100] In this embodiment of the disclosure, the preset dynamic X-ray image training set is configured as a dynamic knee X-ray image training set or a dynamic chest X-ray image training set. Specifically, each dynamic X-ray image in the dynamic knee X-ray image training set or the dynamic chest X-ray image training set is a multi-time knee X-ray image or chest X-ray image of the same subject.

[0101] In this embodiment of the disclosure, those skilled in the art can configure the preset image similarity based on actual needs. For example, the preset image similarity can be configured to 80% or other values ​​less than 100%.

[0102] In this embodiment of the disclosure, the method further includes: determining dynamic X-ray images to be labeled in a set of dynamic X-ray imaging training images using the dynamic X-ray image labeling method described above; labeling the regions to be segmented in the dynamic X-ray images to obtain corresponding mask label images of the regions to be segmented (mask label images); and training a set segmentation network based on the dynamic X-ray images to be labeled and their corresponding mask label images of the regions to be segmented to obtain a segmentation model.

[0103] This disclosure also proposes a method for determining the contact surface length of the femur and patella, comprising: determining the contact surface of the femur and patella in the X-ray image of the knee joint to be processed using the above-described method for determining the contact surface of the femur and patella; and determining the contact surface length of the femur and patella based on the contact surface.

[0104] The step of determining the contact surface length of the femur and patella based on the contact surface includes: calculating multiple non-zero coordinate distances between non-zero coordinates in the contact surface; and determining the maximum distance among the multiple non-zero coordinate distances as the contact surface length of the femur and patella.

[0105] The step of calculating the distances between multiple non-zero coordinates in the contact surface includes: constructing a vector array using the non-zero coordinates in the contact surface; and calculating the distances between the currently traversed non-zero coordinate and other non-zero coordinates by traversing each non-zero coordinate in the vector array.

[0106] Output the length of the femoral and patellar contact surfaces: Using a segmentation model, the femur and patella are segmented from multi-time dynamic knee X-ray images to obtain a femoral region mask (femoral region mask image or femoral region mask code image) `maskimg1` and a patellar region mask `maskimg2` (patellar region mask image or patellar region mask code image). Here, "mask" and "code" are two ways of representing the same technology and have the same meaning. The overlapping region (`overlap`) of the two masks (femoral region mask `maskimg1` and patellar region mask `maskimg2`) is found, and a vector array is constructed using the non-zero coordinates `pts` of the overlapping region. Each non-zero coordinate `xypoints` in the vector array is traversed, and multiple distances between each non-zero coordinate `xypoints` and other non-zero coordinates are calculated. The two non-zero coordinates corresponding to the largest distance among these multiple distances are determined, and the distance between these two non-zero coordinates is configured as the length of the femoral and patellar contact surfaces.

[0107] This disclosure also proposes a method for processing X-ray images of the knee joint, characterized by including: the method for determining the contact surface of the femur and patella as described above; and / or, the method for determining the length of the contact surface of the femur and patella as described above; and, Extract the femoral mask boundary image corresponding to the femoral mask from the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, determine one or both of the first and second boundary lines among the boundary lines on both sides of the femur or femoral cortex; and / or, take the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, determine the first and second boundary lines among the boundary lines on both sides of the femur or femoral cortex; based on the first and second boundary lines, determine the femoral midline; and / or, using the keypoint detection model corresponding to the keypoint detection network, detect the third keypoint position coordinates corresponding to the highest point coordinates of the upper edge of the patella and the fourth keypoint position coordinates corresponding to the lowest point coordinates of the lower edge of the patella in the knee X-ray image to be processed, and determine the third and fourth keypoint position coordinates; based on the third and fourth keypoint position coordinates, determine the patellar length of the knee X-ray image to be processed.

[0108] This disclosure proposes a method for determining the bilateral boundary lines of the femur, comprising: extracting a femoral mask boundary image corresponding to the femoral mask in a femoral region mask image; and determining one or both of the first and second boundary lines among the bilateral boundary lines of the femur or femoral cortex based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image. This addresses at least one of the following technical problems: the lack of an effective quantitative method for determining the bilateral boundary lines and femoral midline, which hinders the auxiliary diagnosis of fractures and dislocations, assessment of joint development and deformities, guidance of surgical planning and reduction, and monitoring of treatment effects and rehabilitation progress; the inherent subjectivity and difficulty in ensuring objectivity in manually determining the bilateral boundary lines and femoral midline; and the increased workload for radiologists due to the already demanding nature of the procedure.

[0109] In this embodiment of the disclosure, the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image includes: eroding the femoral mask in the femoral region mask image to obtain a femoral mask erosion image; subtracting the femoral mask erosion image from the femoral region mask image to obtain the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image.

[0110] In this embodiment of the disclosure, determining one or both of the first and second boundary lines among the boundary lines on both sides of the femur based on the non-zero coordinates of the femur mask boundary of the femur mask boundary image includes: using polar coordinate Hough transform to determine the first boundary line on one side of the femur corresponding to the first straight line (first polar coordinate equation) based on the first polar coordinate equation corresponding to the first preset distance range and the first preset angle range, and the non-zero coordinates of the first side femur mask boundary corresponding to the femur mask boundary image; and / or, using polar coordinate Hough transform to determine the second boundary line on the other side of the femur corresponding to the second straight line (second polar coordinate equation) based on the second polar coordinate equation corresponding to the second preset distance range and the second preset angle range, and the non-zero coordinates of the second side femur mask boundary corresponding to the femur mask boundary image.

[0111] In this embodiment of the disclosure, determining the first preset distance range includes: using the diagonal length of the femoral mask boundary image to determine the first preset distance range corresponding to the first distance from the origin of the rectangular coordinate system corresponding to the first polar coordinate parameter in the first polar coordinate equation to the first straight line.

[0112] In this embodiment of the disclosure, determining the second preset distance range includes: using the diagonal length of the femoral mask boundary image to determine the second preset distance range corresponding to the second distance from the origin of the rectangular coordinate system corresponding to the third polar coordinate parameter in the second polar coordinate equation to the second straight line.

[0113] In this embodiment of the disclosure, the first preset included angle range is configured to be 0 to 180 degrees; the first preset distance range is configured to be -diagonal length to +diagonal length; wherein - and + represent negative and positive signs, respectively.

[0114] In this embodiment of the disclosure, the second preset included angle range is configured to be 0 to 180 degrees and / or, and the second preset distance range is configured to be -diagonal length to +diagonal length; wherein - and + represent negative and positive signs, respectively.

[0115] In this embodiment of the disclosure, determining one or both of the first and second boundary lines among the boundary lines on both sides of the femur based on the non-zero coordinates of the femur mask boundary in the femur mask boundary image includes: determining a first distance range corresponding to the first distance from the origin of the rectangular coordinate system corresponding to the first polar coordinate parameter in the first polar coordinate equation to the first straight line using the diagonal length of the femur mask boundary image; configuring a first angle range corresponding to the first angle between the perpendicular line from the origin of the rectangular coordinate system corresponding to the second polar coordinate parameter in the first polar coordinate equation to the first straight line and the x-axis; and using the polar coordinate Hough transform to determine the boundary lines based on the first distance range and the first angle range corresponding to the first polar coordinate equation and the non-zero coordinates of the first side femur mask boundary corresponding to the femur mask boundary image. First, determine the first boundary line on one side of the femur corresponding to the first straight line (first polar coordinate equation); and / or, determine the second distance range corresponding to the second distance from the origin of the rectangular coordinate system corresponding to the third polar coordinate parameter in the second polar coordinate equation to the second straight line using the diagonal length of the femoral mask boundary image; configure the second angle range corresponding to the second included angle between the perpendicular line from the origin of the rectangular coordinate system corresponding to the fourth polar coordinate parameter in the second polar coordinate equation to the second straight line and the x-axis; based on the second distance range and the second included angle range corresponding to the second polar coordinate equation and the non-zero coordinates of the second side femoral mask boundary corresponding to the femoral mask boundary image, determine the second boundary line on the other side of the femur corresponding to the second straight line (second polar coordinate equation) using polar coordinate Hough transform.

[0116] In this embodiment of the disclosure, the first included angle range is configured to be 0 to 180 degrees; and / or, the first distance range is configured to be -diagonal length to +diagonal length; wherein - and + represent negative and positive signs respectively; and / or, the second included angle range is configured to be 0 to 180 degrees; and / or, the second distance range is configured to be -diagonal length to +diagonal length; wherein - and + represent negative and positive signs respectively.

[0117] In this embodiment of the disclosure, before extracting the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image, the method includes: using a knee X-ray imaging image segmentation model to perform femoral segmentation on the knee X-ray imaging image to be processed, thereby obtaining the femoral region mask image corresponding to the knee X-ray imaging image to be processed.

[0118] In this embodiment of the disclosure, determining the knee X-ray image segmentation model includes: constructing a multi-segmentation region weight loss function based on the mask label images corresponding to a preset knee X-ray image training set or a preset dynamic knee X-ray image training set; wherein, the mask label images include at least a femoral mask label or the mask label images include one or more mask labels such as a femoral mask label, a patellar mask label, a tibial mask label, and a patellar tendon mask label; and training a preset segmentation network using the preset knee X-ray image training set or the preset dynamic knee X-ray image training set and its corresponding mask label images, and the multi-segmentation region weight loss function to obtain the knee X-ray image segmentation model or the dynamic knee X-ray image segmentation model.

[0119] In this embodiment of the disclosure, determining the knee joint X-ray image segmentation model includes: constructing a multi-segmentation region weight loss function based on the mask label images corresponding to a preset knee joint X-ray image training set or a preset dynamic knee joint X-ray image training set; wherein, the mask label images include at least a femoral mask label or the mask label images include one or more mask labels such as a femoral mask label, a patellar mask label, a tibial mask label, and a patellar tendon mask label; and using the preset knee joint X-ray image training set or the preset dynamic knee joint X-ray image training set and its corresponding mask label images, and the multi-segmentation region weight loss function, performing multiple preset segmentation networks... Training is performed to obtain multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models; the multiple knee joint X-ray image segmentation models or the multiple dynamic knee joint X-ray image segmentation models are evaluated to determine the corresponding optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model; based on the optimal knee joint X-ray image segmentation model or the optimal dynamic knee joint X-ray image segmentation model, the knee joint X-ray image or dynamic knee joint X-ray image to be processed is segmented into one or more segments, such as femoral segmentation or segmentation of the femur and patella, tibia, and patellar tendon.

[0120] This disclosure also proposes a method for determining the femoral midline, comprising: using the above-described method for determining the boundary lines on both sides of the femur to determine a first boundary line and a second boundary line among the boundary lines on both sides of the femur or the femoral cortex; and determining the femoral midline based on the first boundary line and the second boundary line.

[0121] In this embodiment of the disclosure, determining the femoral midline based on the first boundary line and the second boundary line includes: determining a first direction vector corresponding to the first boundary line and a second direction vector corresponding to the second boundary line; performing dot products on the first direction vector and the second direction vector with the femoral principal component direction vector corresponding to the femoral region mask image to obtain a first product corresponding to the first direction vector and a second product corresponding to the second direction vector; if the first product is less than a first preset value, inverting the first direction vector in the direction; otherwise, not processing the first direction vector; if the second product is less than a second preset value, inverting the second direction vector in the direction; otherwise, not processing the second direction vector; adding the first direction vector corresponding to the inverted or unprocessed and the second direction vector corresponding to the inverted or unprocessed to obtain the bisecting line direction vector; and determining the femoral midline based on the first boundary line, the second boundary line, and the bisecting line direction vector.

[0122] In this embodiment of the disclosure, determining the femoral midline based on the first boundary line, the second boundary line, and the direction vector of the bisector includes: if the first boundary line and the second boundary line intersect, then determining the femoral midline corresponding to the third polar coordinate equation based on the intersection point, the direction vector of the bisector, and the third polar coordinate equation corresponding to the direction vector of the bisector; otherwise, determining the femoral midline corresponding to the third polar coordinate equation based on the midpoint of the perpendicular segment between the first boundary line and the second boundary line, the direction vector of the bisector, and the third polar coordinate equation corresponding to the direction vector of the bisector.

[0123] In this embodiment of the disclosure, before performing dot products on the first direction vector and the second direction vector with the direction vector corresponding to the femoral region mask image to obtain the first product corresponding to the first direction vector and the second product corresponding to the second direction vector, the method includes: normalizing the first direction vector, the second direction vector and the direction vector corresponding to the femoral region mask image.

[0124] In this embodiment of the disclosure, the values ​​corresponding to the first preset value and the second preset value are the same or different; and / or, the values ​​corresponding to the first preset value and the second preset value are respectively configured to 0.

[0125] In this embodiment of the disclosure, before performing dot products of the first direction vector and the second direction vector with the femoral principal component direction vector corresponding to the femoral region mask image to obtain a first product corresponding to the first direction vector and a second product corresponding to the second direction vector, the method includes: calculating the femoral principal component direction vector corresponding to the non-zero point coordinates of the femoral mask in the femoral region mask image and the center point corresponding to the non-zero point coordinates; using the center point to correct the femoral principal component direction vector to obtain a corrected femoral principal component direction vector; and performing dot products of the first direction vector and the second direction vector with the corrected femoral principal component direction vector corresponding to the femoral region mask image to obtain a first product corresponding to the first direction vector and a second product corresponding to the second direction vector.

[0126] In this embodiment of the disclosure, the step of correcting the femoral principal component direction vector using the center point to obtain a corrected femoral principal component direction vector includes: subtracting the center point from each non-zero point coordinate of the femoral mask in the femoral region mask image to obtain a relative vector of each non-zero point coordinate relative to the center point; multiplying each relative vector by the femoral principal component direction vector of the corresponding non-zero point coordinate to obtain multiple projection values ​​of the femoral principal component direction vector in the current principal component direction; determining the first non-zero point coordinate corresponding to the minimum projection value and the second non-zero point coordinate corresponding to the maximum projection value among the multiple projection values; calculating the first distance and the second distance between the center point of the femoral region mask image and the first non-zero coordinate and the second non-zero coordinate respectively; determining the direction corresponding to the femoral principal component direction vector based on the first distance and the second distance; and correcting the femoral principal component direction vector based on the direction to obtain the corrected femoral principal component direction vector.

[0127] In this embodiment of the disclosure, determining the direction corresponding to the direction vector based on the first distance and the second distance includes: if the first distance is less than the second distance, the direction is from the femoral knee joint to the hip joint; otherwise, the direction is from the hip joint to the femoral knee joint.

[0128] In this embodiment of the disclosure, the step of correcting the femoral principal component direction vector based on the direction to obtain a corrected femoral principal component direction vector includes: if the direction is from the femoral knee joint to the hip joint, then the femoral principal component direction vector is not corrected; if the direction is from the hip joint to the femoral knee joint, then the direction vector is inverted in the direction to obtain a corrected femoral principal component direction vector.

[0129] In this embodiment of the disclosure, before performing dot products on the first direction vector and the second direction vector with the direction vector corresponding to the femoral region mask image to obtain the first product corresponding to the first direction vector and the second product corresponding to the second direction vector, the method includes: normalizing the first direction vector, the second direction vector and the femoral principal component direction vector corresponding to the femoral region mask image.

[0130] In this embodiment of the disclosure, before performing dot products on the first direction vector and the second direction vector with the modified femoral principal component direction vector corresponding to the femoral region mask image to obtain the first product corresponding to the first direction vector and the second product corresponding to the second direction vector, the method includes: normalizing the first direction vector, the second direction vector and the modified femoral principal component direction vector corresponding to the femoral region mask image.

[0131] This disclosure proposes a femoral direction vector correction method, comprising: calculating the femoral principal component direction vector corresponding to the non-zero coordinates of the femoral mask in the femoral region mask image and the center point corresponding to the non-zero coordinates; and correcting the femoral principal component direction vector using the center point to obtain a corrected femoral principal component direction vector. This addresses at least one of the following technical problems: the lack of femoral direction vector correction during the automatic determination of the femoral or femoral cortex boundary lines results in insufficient accuracy for femoral or femoral cortex boundary line detection; the need to avoid femoral midline detection failure; the high subjectivity of manually annotating the tibial boundary line and tibial midline; and the need for further improvement in the accuracy of current automatic detection of tibial boundary lines and tibial midline.

[0132] In this embodiment of the disclosure, calculating the femoral principal component direction vector corresponding to the non-zero coordinates of the femoral mask in the femoral region mask image includes: processing the non-zero coordinates of the femoral mask in the femoral region mask image using a principal component analysis algorithm to obtain the femoral principal component direction vector corresponding to the non-zero coordinates.

[0133] In this embodiment of the disclosure, calculating the center point corresponding to the non-zero coordinates of the femoral mask in the femoral region mask image includes: calculating the average value of the abscissa of the non-zero abscissa and the average value of the ordinate of the non-zero ordinate corresponding to the femoral mask, and determining the center point corresponding to the non-zero coordinates of the femoral mask.

[0134] In this embodiment of the disclosure, the step of correcting the femoral principal component direction vector using the center point to obtain a corrected femoral principal component direction vector includes: subtracting the center point from each non-zero point coordinate of the femoral mask in the femoral region mask image to obtain a relative vector of each non-zero point coordinate relative to the center point; multiplying each relative vector by the femoral principal component direction vector of the corresponding non-zero point coordinate to obtain multiple projection values ​​of the femoral principal component direction vector in the current principal component direction; determining the first non-zero point coordinate corresponding to the minimum projection value and the second non-zero point coordinate corresponding to the maximum projection value among the multiple projection values; calculating the first distance and the second distance between the center point of the femoral region mask image and the first non-zero coordinate and the second non-zero coordinate respectively; determining the direction corresponding to the femoral principal component direction vector based on the first distance and the second distance; and correcting the femoral principal component direction vector based on the direction to obtain the corrected femoral principal component direction vector.

[0135] In this embodiment of the disclosure, determining the direction corresponding to the direction vector based on the first distance and the second distance includes: if the first distance is less than the second distance, the direction is from the femoral knee joint to the femoral hip joint; otherwise, the direction is from the femoral hip joint to the femoral knee joint.

[0136] In this embodiment of the disclosure, the step of correcting the femoral principal component direction vector based on the direction to obtain a corrected femoral principal component direction vector includes: if the direction is from the femoral knee joint to the femoral hip joint, then the femoral principal component direction vector is not corrected; if the direction is from the femoral hip joint to the femoral knee joint, then the femoral principal component direction vector is inverted in the direction to obtain a corrected femoral principal component direction vector.

[0137] In this embodiment of the disclosure, before calculating the femoral principal component direction vector corresponding to the non-zero coordinates of the femoral mask in the femoral region mask image and the center point corresponding to the non-zero coordinates, the method includes: using a knee X-ray imaging image segmentation model to perform femoral segmentation on the knee X-ray imaging image to be processed, thereby obtaining the femoral region mask image corresponding to the knee X-ray imaging image to be processed.

[0138] In this embodiment of the disclosure, determining the knee joint X-ray image segmentation model includes: constructing a multi-segmentation region weight loss function based on the mask label images corresponding to a preset dynamic knee joint X-ray image training set; wherein, the mask label images include at least a femoral mask label or the mask label images include one or more of the following mask labels: femoral mask label, patellar mask label, tibia mask label, and patellar tendon mask label; and training a preset segmentation network using the preset dynamic knee joint X-ray image training set and its corresponding mask label images, and the multi-segmentation region weight loss function to obtain the dynamic knee joint X-ray image segmentation model.

[0139] In this embodiment of the disclosure, determining the knee joint X-ray image segmentation model includes: determining the knee joint X-ray image segmentation model includes: constructing a multi-segmentation region weight loss function based on a preset training set of multiple knee joint X-ray images or a preset dynamic knee joint X-ray image training set corresponding to mask label images; wherein, the mask label images include at least a femoral mask label or the mask label images include one or more mask labels such as a femoral mask label and a patellar mask label, a tibial mask label, and a patellar tendon mask label; using the preset dynamic knee joint X-ray image training set and its corresponding mask label images, the multi-segmentation region weight loss function... The system trains multiple preset segmentation networks to obtain multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models; it evaluates the multiple knee X-ray image segmentation models or the multiple dynamic knee X-ray image segmentation models to determine the optimal knee X-ray image segmentation model or the optimal dynamic knee X-ray image segmentation model; based on the optimal knee X-ray image segmentation model or the optimal dynamic knee X-ray image segmentation model, it performs femoral segmentation or one or more segmentations of the femur, patella, tibia, and patellar tendon on the knee X-ray image or dynamic knee X-ray image to be segmented.

[0140] This disclosure also proposes a method for processing X-ray images of the knee joint, comprising: determining one or two boundary lines, namely a first boundary line and a second boundary line, in the boundary lines on both sides of the femur or femoral cortex, based on the non-zero coordinates of the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; determining a modified femoral principal component direction vector corresponding to the femoral principal component direction vector using the femoral direction vector correction method described above; and determining the femoral midline based on the modified femoral principal component direction vector, the first boundary line, and the second boundary line.

[0141] In this embodiment of the disclosure, determining the femoral midline based on the modified femoral principal component direction vector, the first boundary line, and the second boundary line includes: determining a first direction vector corresponding to the first boundary line and a second direction vector corresponding to the second boundary line; performing dot products on the first direction vector and the second direction vector with the modified femoral principal component direction vector corresponding to the femoral region mask image to obtain a first product corresponding to the first direction vector and a second product corresponding to the second direction vector; if the first product is less than a first preset value, inverting the first direction vector in the direction; otherwise, not processing the first direction vector; if the second product is less than a second preset value, inverting the second direction vector in the direction; otherwise, not processing the second direction vector; adding the first direction vector corresponding to the inverted or unprocessed and the second direction vector corresponding to the inverted or unprocessed to obtain the bisecting line direction vector; and determining the femoral midline based on the first boundary line, the second boundary line, and the bisecting line direction vector.

[0142] Calculate the direction vectors (principal component direction vectors) corresponding to the non-zero coordinates of the femoral mask (femurmask) in the femoral region mask image or femoral region mask image. Specifically, this includes: extracting the non-zero coordinates of the femoral mask (femurmask), calculating the direction vectors (femoral principal component direction vectors) corresponding to the non-zero coordinates of the femoral mask (femoral principal component direction vectors) using the principal component analysis (PCA) algorithm, and normalizing the direction vectors to obtain the normalized direction vector (femoral normalized direction vector) PCA_femurvec. (Note that the direction vector or normalized direction vector PCA_femurvec corresponding to the non-zero coordinates obtained at this time may point from the side of the femur near the patella to the side near the hip joint, or it may point from the side of the femur near the hip joint to the side of the femur near the patella.)

[0143] Calculate the center point center_femur of the femoral mask. Specifically, this includes: calculating the average of the x-coordinates of all non-zero points on the femoral mask and the average of the y-coordinates of all non-zero points on the femoral mask, to obtain the x-coordinate and y-coordinate of the center point center_femur.

[0144] Correcting the direction vectors corresponding to the non-zero coordinates of the femurmask. Specifically, this includes: subtracting the center point `center_femur` from each non-zero coordinate of the femurmask to obtain the relative vector of each non-zero coordinate relative to the center point `center_femur`; multiplying each relative vector by the direction vector or normalized direction vector `PCA_femurvec` of the corresponding non-zero coordinate to obtain multiple projection values ​​`projs_pts` of the direction vector in the current principal component direction; finding the first and second non-zero coordinates of the femurmask corresponding to the minimum and maximum projection values ​​among the multiple projection values ​​`projs_pts`, denoted as `end1_pos` (first non-zero coordinate) and `end2_pos` (second non-zero coordinate). Take the center point `imgpoint_center` of the image to be segmented or the segmentation mask image corresponding to the image to be segmented; calculate the first distance `dist1` between the first non-zero point coordinate `end1_pos` and `imgpoint_center`, and the second distance `dist2` between the second non-zero point coordinate `end2_pos` and `imgpoint_center`; if the first distance `dist1` is less than the second distance `dist2`, then the direction of the direction vector or normalized direction vector `PCA_femurvec` is from the femoral knee joint to the femoral hip joint; if the first distance `dist1` is greater than the second distance `dist2`, then invert the direction of the direction vector or normalized direction vector `PCA_femurvec` (add a negative sign), that is, set the direction vector or normalized direction vector `PCA_femurvec` = -PCA_femurvec. After the above steps, the direction vector corresponding to the non-zero point coordinates of the femoral mask `femurmask` is corrected, and the corrected direction vector is obtained.

[0145] Extract the set of non-zero coordinates of the femoral mask boundary image (femurmask_edge) corresponding to the femoral mask boundary image (femurmask_edge). Specifically, this includes: eroding the femoral mask to obtain the eroded femoral mask image femurmaskerodeimg; subtracting the eroded femoral mask image femurmaskerodeimg from the femoral mask to extract the femoral mask boundary image femurmask_edge; and extracting the set of non-zero coordinates of the femoral mask boundary image femurmask_edge femurmask_edgeptrvec.

[0146] Based on the non-zero coordinate set femurmask_edgeptrvec of the femoral mask boundary, fit the first polar coordinate equation of the first straight line corresponding to the two sides of the femur or femoral cortex and the second polar coordinate equation of the second straight line opposite to the first straight line (the first straight line and the second straight line constitute the boundary of the two sides of the femur). Specifically, this includes: based on the diagonal length of the image to be segmented or the segmentation mask image corresponding to the image to be segmented being max_rho, determining the range of the first polar coordinate parameter (the first distance from the origin of the rectangular coordinate system to the first straight line) ρ in the polar coordinate equation to be from -max_rho to max_rho; configuring the range of the second polar coordinate parameter (the first angle between the perpendicular from the origin of the rectangular coordinate system to the first straight line and the x-axis) θ in the polar coordinate equation to be from 0 to 180 degrees; based on the polar coordinate equations corresponding to the distance range and the angle range and the set of non-zero coordinates of the femoral mask boundary (the set of non-zero coordinates of the first femoral mask boundary and the set of non-zero coordinates of the second femoral mask boundary), using the polar coordinate Hough (line) transformation (the line from a point in polar coordinates to the Hough space), determining the two most likely straight lines, respectively denoted as the first straight line Linerleft (the first boundary line of the femoral side) corresponding to one side of the femur and the second straight line Lineright (the second boundary line of the femoral side) corresponding to the other side of the femur. In this system, each straight line is expressed in polar coordinates as x*cosθ+y*sinθ=ρ, where x and y are the variables corresponding to the polar coordinate equation. Specifically, the first polar coordinate equation for the first straight line Linerleft on one side of the femur is configured as x*cosθ1+y*sinθ1=ρ1, and the second polar coordinate equation for the second straight line Lineright on the other side of the femur is configured as x*cosθ2+y*sinθ2=ρ2. Here, θ1 and θ2 represent the first angle between the perpendicular from the origin of the rectangular coordinate system to the first straight line and the x-axis, and the second angle between the perpendicular from the origin of the rectangular coordinate system to the second straight line and the x-axis, respectively. ρ1 and ρ2 represent the first distance from the origin of the rectangular coordinate system to the first straight line and the second distance from the origin of the rectangular coordinate system to the second straight line, respectively. The first straight line is represented using the first polar coordinate equation; the second straight line is represented using the corresponding second polar coordinate equation.

[0147] Based on the first straight line Linerleft corresponding to one side of the femur and the second straight line Lineright corresponding to the other side of the femur, determine the polar coordinate equation corresponding to the femoral midline. Specifically, this includes: based on the first straight line Linerleft corresponding to one side of the femur and the second straight line Lineright corresponding to the other side of the femur, determine the first direction vector vec1 of the two straight lines as (-sinθ1, cosθ1) and the second direction vector vec2 as (-sinθ2, cosθ2).

[0148] Normalize the first direction vector vec1 and the second direction vector vec2 respectively to obtain a first normalized direction vector and a second normalized direction vector. Then, perform a dot product between the first normalized direction vector, the first normalized direction vector, the second normalized direction vector, or the direction vector corresponding to the non-zero coordinates of the second direction vector and the modified femoral mask (femurmask) to obtain a first product of the first normalized direction vector and a second product of the second normalized direction vector. If the first product is less than 0, invert the first direction vector or the first normalized direction vector in the direction (add a negative sign); if the second product is less than 0, invert the second direction vector or the second normalized direction vector in the direction (add a negative sign).

[0149] After the above steps, ensure that the directions of the first and second direction vectors of the two straight lines (the first straight line Linerleft corresponding to one side of the femur and the second straight line Lineright corresponding to the other side of the femur) are consistent with the direction of the femoral direction vector or the normalized femoral direction vector PCA_femurvec. After determining the directions, add the first direction vector vec1 and the second direction vector vec2 to obtain the bisector direction vector (i.e., the direction vector of the middle bisector corresponding to the first straight line Linerleft corresponding to one side of the femur and the second straight line Lineright corresponding to the other side of the femur). Normalize the bisector direction vector to obtain the normalized bisector direction vector femurvec3, which is (-sinθ3, cosθ3). The polar coordinate equation of the third straight line corresponding to the bisector direction vector or the normalized bisector direction vector femurvec3 is configured as x*cosθ3+y*sinθ3=ρ3. Where θ3 represents the third angle between the perpendicular line from the origin of the rectangular coordinate system to the third line and the x-axis, and ρ3 represents the third distance from the origin of the rectangular coordinate system to the third line. Taking the intersection point (x, y) of the first line Linerleft and the second line Lineright, the third distance ρ3 from the origin of the rectangular coordinate system to the third line can be calculated based on the intersection point (x, y), the bisector direction vector, and the third polar coordinate equation. If the first line Linerleft and the second line Lineright do not intersect, the third distance ρ3 from the origin of the rectangular coordinate system to the third line is determined based on the midpoint (x, y) of the perpendicular segment between the first line Linerleft and the second line Lineright, the bisector direction vector, and the third polar coordinate equation. Therefore, the third polar coordinate equation corresponding to the femoral midline is configured as x*cosθ3 + y*sinθ3 = ρ3.

[0150] The preset keypoint detection network corresponding to the keypoint detection network can be configured as one or more of the following deep learning networks: OpenPose, HRNet (High-Resolution Network), Hourglass Network, DEKR (Distributed KeypointRegression), YOLO-KP, CenterNet, Cornernet, TokenPose, PoseTransformer, Mask R-CNN, etc.

[0151] Before determining the coordinates of the third and fourth key points, the process includes: segmenting the patella on the X-ray image of the knee to be processed; if the patella segmentation fails, the key point detection model corresponding to the key point detection network is used to detect the coordinates of the third key point corresponding to the highest point of the upper edge of the patella and the coordinates of the fourth key point corresponding to the lowest point of the lower edge of the patella in the X-ray image of the knee to be processed, and the coordinates of the third and fourth key points are determined; otherwise, the first position coordinates of one end of the patella and the second position coordinates of the other end of the patella are determined based on the patella mask image corresponding to the X-ray image of the knee to be processed, and the patella length of the X-ray image of the knee to be processed is determined.

[0152] The step of determining the patella length of the knee X-ray image based on the patella mask image corresponding to the knee X-ray image to be processed includes: calculating the distance between the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the second position coordinates of the second position coordinates of the second position coordinates of the knee X-ray image to be processed, and determining the patella length of the knee X-ray image to be processed.

[0153] Determining that the patellar segmentation of the knee X-ray image to be processed has failed includes: performing patellar segmentation on the knee X-ray image to be processed to obtain a patellar mask image; if the similarity between the patellar mask in the patellar mask image and a preset patellar mask is less than a preset similarity, then the patellar segmentation of the knee X-ray image to be processed is determined to have failed; or, if performing patellar segmentation on the knee X-ray image to be processed does not yield a patellar mask image, then the patellar segmentation of the knee X-ray image to be processed has failed; or, if performing patellar segmentation on the knee X-ray image to be processed results in a patellar mask image that lacks the coordinates of the third key point and the fourth key point corresponding to the patella, then the patellar segmentation of the knee X-ray image to be processed has failed.

[0154] Before performing patellar segmentation on the knee X-ray image to be processed, a knee joint X-ray image segmentation model is constructed; the knee joint X-ray image segmentation model is used to perform patellar segmentation on the knee X-ray image to be processed.

[0155] The construction of the knee X-ray image segmentation model includes: training a pre-defined segmentation network based on deep learning using a set of multiple knee X-ray images or a set of dynamic knee X-ray images and their corresponding mask label images to construct the knee X-ray image segmentation model; wherein, the mask label images include: patellar mask label; or, the mask label images include: one or more of femoral mask label, tibial mask label, patellar tendon mask label, and patellar mask label.

[0156] The construction of the knee joint X-ray image segmentation model includes: constructing a multi-segmentation region weight loss function based on the mask label images corresponding to a set of multiple knee joint X-ray images or a set of dynamic knee joint X-ray images; wherein, the mask label images include one or more of the following: patellar mask label, femoral mask label, tibial mask label, and patellar tendon mask label; using the set of multiple knee joint X-ray images or the set of dynamic knee joint X-ray images and their corresponding mask label images, and the multi-segmentation region weight loss function, multiple segmentation regions are segmented... The network is trained to obtain multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models; the multiple knee joint X-ray image segmentation models or the multiple dynamic knee joint X-ray image segmentation models are evaluated to determine the corresponding optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model; based on the optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model, the dynamic knee joint X-ray image to be segmented is segmented into one or more of the patellar tendon, femur, tibia and patella.

[0157] The step of determining the patella length of the knee X-ray image to be processed based on the coordinates of the third key point and the fourth key point includes: calculating the distance between the coordinates of the third key point and the fourth key point to determine the patella length of the knee X-ray image to be processed.

[0158] Constructing the keypoint detection model corresponding to the keypoint detection network includes: obtaining the third keypoint position coordinate labels corresponding to the highest point of the upper edge of the patella and the fourth keypoint position coordinate labels corresponding to the lowest point of the lower edge of the patella from multiple knee X-ray images; training the preset keypoint detection network using the first keypoint position labels and the second keypoint position labels to obtain the keypoint detection model.

[0159] Before training the preset keypoint detection network using the first keypoint location label and the second keypoint location label, the process includes: pre-training the preset keypoint detection network using multiple preset pose points corresponding to human pose images and / or multiple preset facial feature points corresponding to facial images to obtain a pre-trained keypoint detection model; then, re-training the pre-trained keypoint detection model using the first keypoint location label and the second keypoint location label to obtain a keypoint detection model.

[0160] In this embodiment of the disclosure, the patella can be manually delineated in each knee X-ray image of the dynamic knee joint to be processed, so as to determine the patella length in each knee X-ray image of the dynamic knee joint to be processed.

[0161] This disclosure also proposes a dynamic assessment method, comprising: determining the patellar length in each knee X-ray image in a dynamic knee X-ray image to be processed using the patellar length determination method described above; wherein the dynamic knee X-ray image to be processed is a sequence of multiple knee X-ray images taken of the same subject in a leg movement state; determining the patellar length corresponding to each knee X-ray image in the dynamic knee X-ray image to be processed; and performing a dynamic assessment of the subject based on the patellar length and the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray image to be processed.

[0162] The step of determining the patella length corresponding to each knee X-ray image in the dynamic knee X-ray image to be processed includes: using a keypoint detection model corresponding to a keypoint detection network to detect the coordinates of a first keypoint corresponding to the lowest point of the lower edge of the patella and the coordinates of a second keypoint corresponding to the last edge of the patellar tendon insertion point on the tibialis ridge in the knee X-ray image to be processed, and determining the coordinates of the first keypoint and the second keypoint; and determining the patellar tendon length of the knee X-ray image to be processed based on the coordinates of the first keypoint and the second keypoint.

[0163] Similarly, the preset keypoint detection network corresponding to the keypoint detection network can be configured as one or more of the following deep learning networks: OpenPose, HRNet (High-Resolution Network), Hourglass Network, DEKR (Distributed KeypointRegression), YOLO-KP, CenterNet, Cornernet, TokenPose, PoseTransformer, Mask R-CNN, etc.

[0164] Before determining the coordinates of the first key point and the second key point, the process includes: segmenting the patellar tendon in the X-ray image of the knee to be processed; if the patellar tendon segmentation of the X-ray image of the knee to be processed fails, the key point detection model corresponding to the key point detection network is used to detect the coordinates of the first key point corresponding to the lowest point of the patellar lower edge and the second key point corresponding to the last edge of the patellar tendon insertion point on the tibialis fossa in the X-ray image of the knee to be processed, and to determine the coordinates of the first key point and the second key point; otherwise, the first position coordinates of one end of the patellar tendon and the second position coordinates of the other end of the patellar tendon are determined based on the patellar tendon mask image corresponding to the X-ray image of the knee to be processed, and the length of the patellar tendon in the X-ray image of the knee to be processed is determined.

[0165] The step of determining the first position coordinates of one end of the patellar tendon and the second position coordinates of the other end of the patellar tendon based on the patellar tendon mask image corresponding to the knee X-ray image to be processed, and determining the patellar tendon length of the knee X-ray image to be processed, includes: calculating the distance between the first position coordinates of one end of the patellar tendon and the second position coordinates of the other end of the patellar tendon, and determining the patellar tendon length of the knee X-ray image to be processed.

[0166] Determining that the patellar tendon segmentation of the knee X-ray image to be processed has failed includes: performing patellar tendon segmentation on the knee X-ray image to be processed to obtain a patellar tendon mask image; if the similarity between the patellar tendon mask in the patellar tendon mask image and a preset patellar tendon mask is less than a preset similarity, then the patellar tendon segmentation of the knee X-ray image to be processed is determined to have failed; or, if performing patellar tendon segmentation on the knee X-ray image to be processed does not yield a patellar tendon mask image, then the patellar tendon segmentation of the knee X-ray image to be processed has failed; or, if performing patellar tendon segmentation on the knee X-ray image to be processed results in a patellar tendon mask image lacking the coordinates of the first key point and the second key point corresponding to the patellar tendon, then the patellar tendon segmentation of the knee X-ray image to be processed has failed.

[0167] Before detecting the first keypoint coordinates corresponding to the lowest point of the patella and the second keypoint coordinates corresponding to the last edge of the patellar tendon insertion in the knee X-ray image to be processed, or before the patellar tendon segmentation of the knee X-ray image to be processed fails, the method further includes: if the patellar tendon segmentation of the knee X-ray image to be processed fails, then performing patellar and tibial segmentation on the knee X-ray image to be processed; if the patellar and / or tibial segmentation of the knee X-ray image to be processed fails, then using the keypoint detection model corresponding to the keypoint detection network, the method further includes: if the patellar tendon segmentation of the knee X-ray image to be processed fails, then performing patellar and tibial segmentation on the knee X-ray image to be processed; if the patellar and / or tibial segmentation of the knee X-ray image to be processed fails, then using the keypoint detection network corresponding to the keypoint detection model, the method further includes: if the patellar tendon segmentation of the knee X-ray image to be processed fails, then performing patellar and tibial segmentation on the knee X-ray image to be processed. The key point detection model detects the coordinates of the first key point corresponding to the lowest point of the patellar lower edge and the second key point corresponding to the coordinates of the last edge of the patellar tendon insertion point on the tibia in the X-ray image of the knee to be processed; otherwise, it determines the coordinates of the first key point based on the patellar mask image corresponding to the X-ray image of the knee to be processed; it determines the coordinates of the second key point based on the tibiar mask image corresponding to the X-ray image of the knee to be processed; and it determines the length of the patellar tendon in the X-ray image of the knee to be processed based on the coordinates of the first key point and the second key point.

[0168] Determining the coordinates of the first key point based on the patellar mask image corresponding to the knee X-ray image to be processed includes: performing edge detection on the patellar mask image to obtain a patellar mask edge image; determining the lowest point of the patellar mask edge in the patellar mask edge image as the coordinates of the first key point; or, determining the patellar mask corresponding to the lowest position among multiple patellar masks in the patellar region mask image (patellar mask image) corresponding to the knee X-ray image to be processed; and determining the patellar region mask corresponding to the lowest position as the coordinates of the lowest point of the lower edge of the patella corresponding to the knee X-ray image to be processed.

[0169] The step of determining the coordinates of the second key point based on the tibial mask image corresponding to the knee X-ray image to be processed includes: determining the third and fourth boundary lines among the boundary lines on both sides of the tibia or tibial cortex according to the non-zero coordinates of the tibial mask boundary in the tibial mask boundary image corresponding to the tibial mask image; selecting the reference boundary line closest to the patellar mask in the patellar mask image from the third and fourth boundary lines; determining whether the reference boundary line has a common mask point with the patellar mask in the patellar mask image; if a common mask point exists, taking any mask point among the common mask points as the starting point, and following the direction of the reference boundary line, determining the first intersection point corresponding to the tibial mask in the tibial mask image or the tibial mask boundary in the tibial mask boundary image corresponding to the tibial mask image; and configuring the first intersection point as the coordinates of the second key point.

[0170] The step of determining the second key point location coordinates based on the tibial mask image corresponding to the knee X-ray image to be processed further includes: if there is no common mask point, determining the reference boundary point closest to the patellar mask edge in the patellar mask edge image corresponding to the reference boundary line as the starting point; using the starting point as the starting coordinates, determining the first intersection point with the tibial mask in the tibial mask image or the tibial mask boundary in the tibial mask boundary image corresponding to the tibial mask image according to the direction of the reference boundary line; and configuring the first intersection point as the second key point location coordinates.

[0171] Constructing the keypoint detection model corresponding to the keypoint detection network includes: obtaining the first keypoint position coordinate labels corresponding to the coordinates of the lowest point of the lower edge of the patella and the second keypoint position coordinate labels corresponding to the coordinates of the last edge of the patellar tendon insertion point on the tibia of multiple knee X-ray images; training the preset keypoint detection network using the first keypoint position labels and the second keypoint position labels to obtain the keypoint detection model.

[0172] Before training the preset keypoint detection network using the first keypoint location label and the second keypoint location label, the process includes: pre-training the preset keypoint detection network using multiple preset pose points corresponding to human pose images and / or multiple preset facial feature points corresponding to facial images to obtain a pre-trained keypoint detection model; then, re-training the pre-trained keypoint detection model using the first keypoint location label and the second keypoint location label to obtain a keypoint detection model.

[0173] The step of determining the patellar tendon length of the knee X-ray image to be processed based on the coordinates of the first key point and the coordinates of the second key point includes: calculating the distance between the coordinates of the first key point and the coordinates of the second key point to determine the patellar tendon length of the knee X-ray image to be processed.

[0174] The dynamic evaluation of the subject's patellar tendon based on the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed includes: calculating the difference in patellar tendon length between adjacent knee X-ray images in the dynamic knee X-ray images to be processed; and using the difference in patellar tendon length between adjacent times and the corresponding preset length difference between adjacent times to dynamically evaluate the subject's patellar tendon.

[0175] The method of dynamically assessing the patellar tendon of the examinee by using the patellar tendon length difference between adjacent time points and the corresponding preset length difference between adjacent time points includes: if the patellar tendon length difference between adjacent time points is less than the corresponding preset length difference between adjacent time points, then assessing that the patellar tendon movement at the adjacent time point is abnormal; otherwise, assessing that the patellar tendon movement at the adjacent time point is normal.

[0176] The dynamic assessment of the subject's patellar tendon based on the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed includes: determining the patellar region mask image corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed; calculating the longest diagonal length of the patella corresponding to the patellar region mask image of each knee X-ray image; calculating the length ratio between the patellar tendon length and the longest diagonal length of the patella in each knee X-ray image; and assessing the subject's patella based on the length ratio corresponding to each knee X-ray image and a preset length ratio.

[0177] The patellar assessment of the subject based on the length ratio corresponding to each knee X-ray image and a preset length ratio includes: if the length ratio corresponding to a first moment in each knee X-ray image is less than a first preset length ratio, then the subject's patella at the first moment is determined to be a low patella; if the length ratio corresponding to a second moment in each knee X-ray image is greater than a second preset length ratio, then the subject's patella at the second moment is determined to be a high patella; if the length ratio corresponding to a third moment in each knee X-ray image is between the second preset length ratio and the second preset length ratio, then the subject's patella at the second moment is determined to be in a normal state. Wherein, the second preset length ratio is greater than the first preset length ratio; the first preset length ratio can be configured to 0.8 or other values; the second preset length ratio can be configured to 1.2 or other values.

[0178] This disclosure proposes a method for determining the joint angle between the femur and tibia, comprising: determining the joint angle between the femoral midline and the tibial midline based on the femoral midline and tibial midline corresponding to the X-ray image of the knee joint to be processed; and calculating the joint angle between the femur and tibia based on the joint angle, a third-direction vector corresponding to the femoral midline, and a sixth-direction vector corresponding to the tibial midline. This addresses the current technical problem of not being able to accurately determine the joint angle, thus affecting subsequent dynamic evaluation of the joint angle.

[0179] In this embodiment of the disclosure, annotations can be manually made on the X-ray image of the knee joint to be processed to determine the femoral midline and tibial midline corresponding to the X-ray image of the knee joint to be processed. Alternatively, a processing algorithm can be used to process the X-ray image of the knee joint to be processed to determine the femoral midline and tibial midline corresponding to the X-ray image of the knee joint to be processed.

[0180] In this embodiment of the disclosure, determining the joint angle between the femoral midline and the tibial midline based on the femoral midline and tibial midline corresponding to the X-ray image of the knee joint to be processed includes: establishing a system of polar coordinate equations based on the femoral midline and tibial midline corresponding to the X-ray image of the knee joint to be processed; and using the system of polar coordinate equations to determine the joint angle between the femoral midline and the tibial midline.

[0181] In this embodiment of the disclosure, determining the joint angle point between the femoral midline and the tibial midline using the polar coordinate equations includes: if the polar coordinate equations have a unique solution, determining whether the coordinates corresponding to the unique solution are within the X-ray image of the knee joint to be processed; if they are within the X-ray image of the knee joint to be processed, determining the coordinates corresponding to the unique solution as the joint angle point between the femoral midline and the tibial midline; if they are not within the X-ray image of the knee joint to be processed or the polar coordinate equations have no solution, then based on the femoral midline and the femoral mask boundary image... The first coordinate point corresponding to the joint angle is determined by using the first non-zero coordinate corresponding to the boundary of the femoral mask, the coordinates of the femoral midline, and the center point of the X-ray image of the knee joint to be processed; the second coordinate point corresponding to the joint angle is determined by using the second non-zero coordinate corresponding to the boundary of the tibial mask in the image of the tibial midline and the coordinates of the center point of the X-ray image of the knee joint to be processed; the midpoint between the first coordinate point and the second coordinate point is calculated, and the midpoint is determined as the joint angle between the femoral midline and the tibial midline.

[0182] In this embodiment of the disclosure, determining the first coordinate point corresponding to the joint corner point based on the first non-zero coordinates corresponding to the femoral mask boundary in the femoral midline and femoral mask boundary image, the coordinates of the center point of the femoral midline and the X-ray image of the knee joint to be processed, includes: calculating the first central axis distance from each non-zero coordinate in the first non-zero coordinates to the femoral midline based on the first non-zero coordinates corresponding to the femoral mask boundary in the femoral midline and femoral mask boundary image; determining a set of selected femoral mask boundary non-zero coordinates whose first central axis distance from each non-zero coordinate in the first non-zero coordinates to the femoral midline is within a preset pixel size length; calculating the first center distance from each selected femoral mask boundary non-zero coordinate in the set of selected femoral mask boundary non-zero coordinates to the center point coordinates of the X-ray image of the knee joint to be processed; and configuring the selected femoral mask boundary non-zero coordinate corresponding to the smallest center distance among the multiple first center distances as the first coordinate point.

[0183] In this embodiment of the disclosure, determining the second coordinate point corresponding to the joint corner point based on the second non-zero coordinates corresponding to the tibial mask boundary in the tibial midline and tibial mask boundary image, the tibial midline, and the center point coordinates of the X-ray image of the knee joint to be processed includes: calculating the second central axis distance from each non-zero coordinate in the second non-zero coordinates to the tibial midline based on the second non-zero coordinates corresponding to the tibial mask boundary in the tibial midline and tibial mask boundary image; determining a set of selected tibial mask boundary non-zero coordinates whose distance from each non-zero coordinate in the second non-zero coordinates to the tibial midline is within a preset pixel size length; calculating the second center distance from each selected tibial mask boundary non-zero coordinate in the set of selected tibial mask boundary non-zero coordinates to the center point coordinates of the X-ray image of the knee joint to be processed; and configuring the selected tibial mask boundary non-zero coordinate corresponding to the smallest center distance among multiple second center distances as the second coordinate point.

[0184] In this embodiment of the disclosure, determining the femoral midline includes: extracting the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; determining the first boundary line and the second boundary line among the boundary lines on both sides of the femur or femoral cortex based on the non-zero coordinates of the femoral mask boundary image; and determining the femoral midline based on the first boundary line and the second boundary line.

[0185] In this embodiment of the disclosure, determining the first boundary line and the second boundary line in the boundary lines on both sides of the femur or femoral cortex includes: determining the first direction vector corresponding to the first boundary line and the second direction vector corresponding to the second boundary line; performing dot products on the first direction vector and the second direction vector with the femoral principal component direction vector corresponding to the femoral region mask image to obtain the first product corresponding to the first direction vector and the second product corresponding to the second direction vector; if the first product is less than a first preset value, inverting the first direction vector in the direction; otherwise, not processing the first direction vector; if the second product is less than a second preset value, inverting the second direction vector in the direction; otherwise, not processing the second direction vector; adding the first direction vector corresponding to the inverted or unprocessed and the second direction vector corresponding to the inverted or unprocessed to obtain the bisecting line direction vector; and determining the femoral midline based on the first boundary line, the second boundary line, and the bisecting line direction vector.

[0186] In this embodiment of the disclosure, determining the femoral midline based on the first boundary line, the second boundary line, and the direction vector of the bisector includes: if the first boundary line and the second boundary line intersect, then determining the femoral midline corresponding to the third polar coordinate equation based on the intersection point, the direction vector of the bisector, and the third polar coordinate equation corresponding to the direction vector of the bisector; otherwise, determining the femoral midline corresponding to the third polar coordinate equation based on the midpoint of the perpendicular segment between the first boundary line and the second boundary line, the direction vector of the bisector, and the third polar coordinate equation corresponding to the direction vector of the bisector.

[0187] In this embodiment of the disclosure, determining the tibial midline includes: extracting the tibial mask boundary image corresponding to the tibial mask in the tibial region mask image; determining the third boundary line and the fourth boundary line among the boundary lines on both sides of the tibia or tibial cortex based on the tibial mask boundary image and the non-zero coordinates of the tibial mask boundary of the tibial mask boundary image; and determining the tibial midline based on the third boundary line and the fourth boundary line.

[0188] In this embodiment of the disclosure, determining the tibial midline based on the third boundary line and the fourth boundary line includes: determining a fourth direction vector corresponding to the third boundary line and a fifth direction vector corresponding to the fourth boundary line; performing dot products on the fourth direction vector and the fifth direction vector with the tibial principal component direction vector corresponding to the tibial region mask image to obtain a third product corresponding to the fourth direction vector and a fourth product corresponding to the fifth direction vector; if the third product is less than a third preset value, inverting the fourth direction vector in the direction; otherwise, not processing the fourth direction vector; if the fourth product is less than a fourth preset value, inverting the fifth direction vector in the direction; otherwise, not processing the fifth direction vector; adding the inverted or unprocessed fourth direction vector and the inverted or unprocessed fifth direction vector to obtain the bisector direction vector; and determining the tibial midline based on the third boundary line, the fourth boundary line, and the bisector direction vector.

[0189] In this embodiment of the disclosure, determining the tibial midline based on the third boundary line, the fourth boundary line, and the bisector direction vector includes: if the third boundary line and the fourth boundary line intersect, then determining the tibial midline corresponding to the sixth polar coordinate equation based on the intersection point, the bisector direction vector, and the sixth polar coordinate equation corresponding to the bisector direction vector; otherwise, determining the tibial midline corresponding to the sixth polar coordinate equation based on the midpoint of the perpendicular segment between the third boundary line and the fourth boundary line, the bisector direction vector, and the sixth polar coordinate equation corresponding to the bisector direction vector.

[0190] In this embodiment of the disclosure, the step of calculating the joint angle between the femur and tibia based on the joint angle point, the third direction vector corresponding to the femoral midline, and the sixth direction vector corresponding to the tibial midline includes: calculating the joint angle between the femur and tibia based on the dot product formula using the third direction vector corresponding to the joint angle point, the femoral midline, and the sixth direction vector corresponding to the tibial midline.

[0191] In this embodiment of the disclosure, the step of calculating the joint angle between the femur and tibia based on the dot product formula, using the joint angle point, the third direction vector corresponding to the femoral midline, and the sixth direction vector corresponding to the tibial midline, includes: normalizing the third direction vector corresponding to the femoral midline and the sixth direction vector corresponding to the tibial midline to obtain a third unit direction vector and a sixth unit direction vector; and calculating the joint angle between the femur and tibia based on the dot product formula, using the joint angle point, the third unit direction vector, and the sixth unit direction vector.

[0192] In this embodiment of the disclosure, the method further includes displaying the joint angle between the femur and tibia.

[0193] In this embodiment of the disclosure, displaying the joint angle between the femur and tibia includes: taking the joint angle point as the starting point, drawing a first ray and a second ray along the third direction vector corresponding to the femoral midline and the sixth direction vector corresponding to the tibial midline, respectively; and displaying the first ray, the second ray, and the joint angle between the first ray and the second ray on the X-ray image of the knee joint to be processed.

[0194] In this embodiment of the disclosure, displaying the joint angle between the femur and tibia further includes: determining the femoral stopping point where the first ray intersects the boundary of the X-ray image of the knee joint to be processed and the tibial stopping point where the second ray intersects the boundary of the X-ray image of the knee joint to be processed; and configuring corresponding first arrows and second arrows for the femoral stopping point and the tibial stopping point, respectively.

[0195] This disclosure also proposes a method for dynamic evaluation of joint angles, comprising: determining the joint angle corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed using the joint angle determination method between the femur and tibia as described above; wherein the dynamic knee X-ray images to be processed are a sequence of multiple knee X-ray images taken of the same subject in a leg movement state; and performing dynamic evaluation of the joint angles of the subject based on the joint angles corresponding to each knee X-ray image.

[0196] In this embodiment of the disclosure, the dynamic evaluation of the joint angle of the subject based on the joint angle corresponding to each knee X-ray image includes: determining the range of knee flexion angle and / or extension angle of the subject based on the maximum and minimum angles among the multiple time-series angles corresponding to the joint angles of each knee X-ray image.

[0197] The embodiment of this disclosure describes a dynamic assessment of the joint angles of the subject based on the joint angles corresponding to each knee X-ray image, which further includes: if the subject's knee flexion angle range is less than a preset joint flexion angle range, then it is determined that the subject has a flexion impairment; and / or, if the subject's knee extension angle range is less than a preset joint extension angle range, then it is determined that the subject has an extension impairment.

[0198] This disclosure also proposes a method for determining the trajectory of joint angle changes, comprising: determining the femoral midline and tibial midline corresponding to the initial joint angle among the multiple time-point joint angles; configuring the initial time joint angle corresponding to the initial time joint angle as an axis point; configuring the femoral midline or tibial midline corresponding to the initial time joint angle as a fixed axis line corresponding to the axis point; wherein the coordinate positions of the fixed axis line and the axis point remain unchanged; and rotating the fixed axis line to the tibial midline or femoral midline corresponding to the other time-point angle based on the multiple time-point angles or other time-point angles besides the initial time-point angle, thereby determining the trajectory of joint angle changes. This addresses the current lack of an effective quantitative method for determining the trajectory of knee joint angle changes, which hinders the auxiliary diagnosis of knee joint diseases, assessment of joint degeneration, optimization of athletic performance, guidance of rehabilitation training, and elucidation of joint movement mechanisms.

[0199] For example, the initial joint angle point corresponding to the initial joint angle is configured as the pivot point; the femoral midline corresponding to the initial joint angle point is configured as the fixed axis corresponding to the pivot point; the coordinate positions of the fixed axis and the pivot point are kept unchanged; based on the multiple time angles or other time angles in the multiple time angles except the initial time angle, the fixed axis is rotated to the tibial midline corresponding to other time angles using the pivot point to determine the joint angle change trajectory corresponding to the examinee.

[0200] For example, the initial joint angle point corresponding to the initial joint angle is configured as the pivot point; the tibial midline corresponding to the initial joint angle point is configured as the fixed axis corresponding to the pivot point; the coordinate positions of the fixed axis and the pivot point are kept unchanged; based on the multiple time angles or other time angles among the multiple time angles excluding the initial time angle, the fixed axis is rotated to the femoral midline corresponding to other time angles using the pivot point to determine the joint angle change trajectory corresponding to the examinee.

[0201] A method for determining patellar tendon length includes: Step S101: Using a keypoint detection model corresponding to a keypoint detection network, detecting the coordinates of a first keypoint corresponding to the lowest point of the patella's lower edge and the coordinates of a second keypoint corresponding to the last edge of the patellar tendon's tibial insertion in the X-ray image of the knee to be processed, and determining the coordinates of the first and second keypoints; Step S102: Based on the coordinates of the first and second keypoints, determining the patellar tendon length of the X-ray image of the knee to be processed. This method aims to solve at least one of the technical problems currently facing the inability to determine patellar tendon length or the inaccuracy of existing techniques in determining patellar tendon length, which hinders the assessment of patellar position and stability, the assistance in diagnosing patellar instability and related diseases, the guidance of surgical treatment and rehabilitation, and the study of knee joint biomechanics.

[0202] Step S101: Using the key point detection model corresponding to the key point detection network, detect the first key point position coordinates corresponding to the lowest point coordinates of the lower edge of the patella and the second key point position coordinates corresponding to the last edge coordinates of the patellar tendon insertion point on the knee X-ray image to be processed, and determine the first key point position coordinates and the second key point position coordinates.

[0203] The preset keypoint detection network corresponding to the keypoint detection network can be configured as one or more of the following deep learning networks: OpenPose, HRNet (High-Resolution Network), Hourglass Network, DEKR (Distributed KeypointRegression), YOLO-KP, CenterNet, Cornernet, TokenPose, PoseTransformer, Mask R-CNN, etc.

[0204] Before determining the coordinates of the first key point and the second key point, the process includes: segmenting the patellar tendon in the X-ray image of the knee to be processed; if the patellar tendon segmentation of the X-ray image of the knee to be processed fails, the key point detection model corresponding to the key point detection network is used to detect the coordinates of the first key point corresponding to the lowest point of the patellar lower edge and the second key point corresponding to the last edge of the patellar tendon insertion point on the tibialis fossa in the X-ray image of the knee to be processed, and to determine the coordinates of the first key point and the second key point; otherwise, the first position coordinates of one end of the patellar tendon and the second position coordinates of the other end of the patellar tendon are determined based on the patellar tendon mask image corresponding to the X-ray image of the knee to be processed, and the length of the patellar tendon in the X-ray image of the knee to be processed is determined.

[0205] The step of determining the first position coordinates of one end of the patellar tendon and the second position coordinates of the other end of the patellar tendon based on the patellar tendon mask image corresponding to the knee X-ray image to be processed, and determining the patellar tendon length of the knee X-ray image to be processed, includes: calculating the distance between the first position coordinates of one end of the patellar tendon and the second position coordinates of the other end of the patellar tendon, and determining the patellar tendon length of the knee X-ray image to be processed.

[0206] Determining that the patellar tendon segmentation of the knee X-ray image to be processed has failed includes: performing patellar tendon segmentation on the knee X-ray image to be processed to obtain a patellar tendon mask image; if the similarity between the patellar tendon mask in the patellar tendon mask image and a preset patellar tendon mask is less than a preset similarity, then the patellar tendon segmentation of the knee X-ray image to be processed is determined to have failed; or, if performing patellar tendon segmentation on the knee X-ray image to be processed does not yield a patellar tendon mask image, then the patellar tendon segmentation of the knee X-ray image to be processed has failed; or, if performing patellar tendon segmentation on the knee X-ray image to be processed results in a patellar tendon mask image lacking the coordinates of the first key point and the second key point corresponding to the patellar tendon, then the patellar tendon segmentation of the knee X-ray image to be processed has failed.

[0207] Before detecting the first keypoint coordinates corresponding to the lowest point of the patella and the second keypoint coordinates corresponding to the last edge of the patellar tendon insertion in the knee X-ray image to be processed, or before the patellar tendon segmentation of the knee X-ray image to be processed fails, the method further includes: if the patellar tendon segmentation of the knee X-ray image to be processed fails, then performing patellar and tibial segmentation on the knee X-ray image to be processed; if the patellar and / or tibial segmentation of the knee X-ray image to be processed fails, then using the keypoint detection model corresponding to the keypoint detection network, the method further includes: if the patellar tendon segmentation of the knee X-ray image to be processed fails, then performing patellar and tibial segmentation on the knee X-ray image to be processed; if the patellar and / or tibial segmentation of the knee X-ray image to be processed fails, then using the keypoint detection network corresponding to the keypoint detection model, the method further includes: if the patellar tendon segmentation of the knee X-ray image to be processed fails, then performing patellar and tibial segmentation on the knee X-ray image to be processed. The key point detection model detects the coordinates of the first key point corresponding to the lowest point of the patellar lower edge and the second key point corresponding to the coordinates of the last edge of the patellar tendon insertion point on the tibia in the X-ray image of the knee to be processed; otherwise, it determines the coordinates of the first key point based on the patellar mask image corresponding to the X-ray image of the knee to be processed; it determines the coordinates of the second key point based on the tibiar mask image corresponding to the X-ray image of the knee to be processed; and it determines the length of the patellar tendon in the X-ray image of the knee to be processed based on the coordinates of the first key point and the second key point.

[0208] Determining the coordinates of the first key point based on the patellar mask image corresponding to the knee X-ray image to be processed includes: performing edge detection on the patellar mask image to obtain a patellar mask edge image; determining the lowest point of the patellar mask edge in the patellar mask edge image as the coordinates of the first key point; or, determining the patellar mask corresponding to the lowest position among multiple patellar masks in the patellar region mask image (patellar mask image) corresponding to the knee X-ray image to be processed; and determining the patellar region mask corresponding to the lowest position as the coordinates of the lowest point of the lower edge of the patella corresponding to the knee X-ray image to be processed.

[0209] The step of determining the coordinates of the second key point based on the tibial mask image corresponding to the knee X-ray image to be processed includes: determining the third and fourth boundary lines among the boundary lines on both sides of the tibia or tibial cortex according to the non-zero coordinates of the tibial mask boundary in the tibial mask boundary image corresponding to the tibial mask image; selecting the reference boundary line closest to the patellar mask in the patellar mask image from the third and fourth boundary lines; determining whether the reference boundary line has a common mask point with the patellar mask in the patellar mask image; if a common mask point exists, taking any mask point among the common mask points as the starting point, and following the direction of the reference boundary line, determining the first intersection point corresponding to the tibial mask in the tibial mask image or the tibial mask boundary in the tibial mask boundary image corresponding to the tibial mask image; and configuring the first intersection point as the coordinates of the second key point.

[0210] The step of determining the second key point location coordinates based on the tibial mask image corresponding to the knee X-ray image to be processed further includes: if there is no common mask point, determining the reference boundary point closest to the patellar mask edge in the patellar mask edge image corresponding to the reference boundary line as the starting point; using the starting point as the starting coordinates, determining the first intersection point with the tibial mask in the tibial mask image or the tibial mask boundary in the tibial mask boundary image corresponding to the tibial mask image according to the direction of the reference boundary line; and configuring the first intersection point as the second key point location coordinates.

[0211] Constructing the keypoint detection model corresponding to the keypoint detection network includes: obtaining the first keypoint position coordinate labels corresponding to the coordinates of the lowest point of the lower edge of the patella and the second keypoint position coordinate labels corresponding to the coordinates of the last edge of the patellar tendon insertion point on the tibia of multiple knee X-ray images; training the preset keypoint detection network using the first keypoint position labels and the second keypoint position labels to obtain the keypoint detection model.

[0212] Before training the preset keypoint detection network using the first keypoint location label and the second keypoint location label, the process includes: pre-training the preset keypoint detection network using multiple preset pose points corresponding to human pose images and / or multiple preset facial feature points corresponding to facial images to obtain a pre-trained keypoint detection model; then, re-training the pre-trained keypoint detection model using the first keypoint location label and the second keypoint location label to obtain a keypoint detection model.

[0213] Step S102: Determine the patellar tendon length of the knee X-ray image to be processed based on the coordinates of the first key point and the second key point.

[0214] The step of determining the patellar tendon length of the knee X-ray image to be processed based on the coordinates of the first key point and the coordinates of the second key point includes: calculating the distance between the coordinates of the first key point and the coordinates of the second key point to determine the patellar tendon length of the knee X-ray image to be processed.

[0215] Output patellar tendon length: Using the HRNet (High-Resolution Network) key point detection network, the coordinates of the lowest point of the lower edge of the patella and the coordinates of the last edge of the patellar tendon insertion point on the tibia are detected, and the distance between the two points is calculated as the patellar tendon length.

[0216] Obtain the first and second key point location labels of the patellar tendon ends corresponding to dynamic knee X-ray images at multiple time points; wherein the first and second key point location labels are configured as the coordinates of the lowest point of the lower edge of the patella and the coordinates of the last edge of the tibial insertion of the patellar tendon, respectively; train a preset key point detection network using the first and second key point location labels to obtain a key point detection model; use the key point detection model to detect the first and second key point locations of the knee X-ray image corresponding to the patellar tendon length to be determined, and obtain the coordinates of the first and second key point locations; based on the coordinates of the first and second key point locations, determine the patellar tendon length of the knee X-ray image corresponding to the patellar tendon length to be determined.

[0217] The step of determining the patellar tendon length of the knee X-ray image corresponding to the patellar tendon length to be determined based on the coordinates of the first key point and the second key point to determine the patellar tendon length of the knee X-ray image corresponding to the patellar tendon length to be determined includes: calculating the distance between the coordinates of the first key point and the coordinates of the second key point to determine the patellar tendon length of the knee X-ray image corresponding to the patellar tendon length to be determined.

[0218] Before training the preset keypoint detection network using the first keypoint location label and the second keypoint location label to obtain the keypoint detection model, the process includes: pre-training the preset keypoint detection network using multiple preset pose points corresponding to human pose images and / or multiple preset facial feature points corresponding to facial images to obtain a pre-trained keypoint detection model; then, re-training the pre-trained keypoint detection model using the first keypoint location label and the second keypoint location label to obtain the keypoint detection model.

[0219] This disclosure also proposes a dynamic evaluation method, comprising: determining the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed using the patellar tendon length method described above; wherein the dynamic knee X-ray images to be processed are a sequence of multiple knee X-ray images taken of the same subject in a leg movement state; and dynamically evaluating the subject's patellar tendon based on the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed.

[0220] The dynamic evaluation of the subject's patellar tendon based on the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed includes: calculating the difference in patellar tendon length between adjacent knee X-ray images in the dynamic knee X-ray images to be processed; and using the difference in patellar tendon length between adjacent times and the corresponding preset length difference between adjacent times to dynamically evaluate the subject's patellar tendon.

[0221] The method of dynamically assessing the patellar tendon of the examinee by using the patellar tendon length difference between adjacent time points and the corresponding preset length difference between adjacent time points includes: if the patellar tendon length difference between adjacent time points is less than the corresponding preset length difference between adjacent time points, then assessing that the patellar tendon movement at the adjacent time point is abnormal; otherwise, assessing that the patellar tendon movement at the adjacent time point is normal.

[0222] The dynamic assessment of the subject's patellar tendon based on the patellar tendon length corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed includes: determining the patellar region mask image corresponding to each knee X-ray image in the dynamic knee X-ray images to be processed; calculating the longest diagonal length of the patella corresponding to the patellar region mask image of each knee X-ray image; calculating the length ratio between the patellar tendon length and the longest diagonal length of the patella in each knee X-ray image; and assessing the subject's patella based on the length ratio corresponding to each knee X-ray image and a preset length ratio.

[0223] The patellar assessment of the subject based on the length ratio corresponding to each knee X-ray image and a preset length ratio includes: if the length ratio corresponding to a first moment in each knee X-ray image is less than a first preset length ratio, then the subject's patella at the first moment is determined to be a low patella; if the length ratio corresponding to a second moment in each knee X-ray image is greater than a second preset length ratio, then the subject's patella at the second moment is determined to be a high patella; if the length ratio corresponding to a third moment in each knee X-ray image is between the second preset length ratio and the second preset length ratio, then the subject's patella at the second moment is determined to be in a normal state. Wherein, the second preset length ratio is greater than the first preset length ratio; the first preset length ratio can be configured to 0.8 or other values; the second preset length ratio can be configured to 1.2 or other values.

[0224] The determination of the patella length in each knee X-ray image of the dynamic knee joint to be processed includes: using a keypoint detection model corresponding to a keypoint detection network to detect the coordinates of the third keypoint corresponding to the highest point of the upper edge of the patella and the fourth keypoint corresponding to the lowest point of the lower edge of the patella in the knee X-ray image to be processed, and determining the coordinates of the third keypoint and the fourth keypoint; and determining the patella length of the knee X-ray image to be processed based on the coordinates of the third keypoint and the fourth keypoint.

[0225] Before determining the coordinates of the third and fourth key points, the process includes: segmenting the patella on the X-ray image of the knee to be processed; if the patella segmentation fails, the key point detection model corresponding to the key point detection network is used to detect the coordinates of the third key point corresponding to the highest point of the upper edge of the patella and the coordinates of the fourth key point corresponding to the lowest point of the lower edge of the patella in the X-ray image of the knee to be processed, and the coordinates of the third and fourth key points are determined; otherwise, the first position coordinates of one end of the patella and the second position coordinates of the other end of the patella are determined based on the patella mask image corresponding to the X-ray image of the knee to be processed, and the patella length of the X-ray image of the knee to be processed is determined.

[0226] The step of determining the patella length of the knee X-ray image based on the patella mask image corresponding to the knee X-ray image to be processed includes: calculating the distance between the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the first position coordinates of the second position coordinates of the second position coordinates of the second position coordinates of the knee X-ray image to be processed, and determining the patella length of the knee X-ray image to be processed.

[0227] Determining that the patellar segmentation of the knee X-ray image to be processed has failed includes: performing patellar segmentation on the knee X-ray image to be processed to obtain a patellar mask image; if the similarity between the patellar mask in the patellar mask image and a preset patellar mask is less than a preset similarity, then the patellar segmentation of the knee X-ray image to be processed is determined to have failed; or, if performing patellar segmentation on the knee X-ray image to be processed does not yield a patellar mask image, then the patellar segmentation of the knee X-ray image to be processed has failed; or, if performing patellar segmentation on the knee X-ray image to be processed results in a patellar mask image that lacks the coordinates of the third key point and the fourth key point corresponding to the patella, then the patellar segmentation of the knee X-ray image to be processed has failed.

[0228] Before performing patellar segmentation on the knee X-ray image to be processed, a knee joint X-ray image segmentation model is constructed; the knee joint X-ray image segmentation model is used to perform patellar segmentation on the knee X-ray image to be processed.

[0229] The construction of the knee X-ray image segmentation model includes: training a pre-defined segmentation network based on deep learning using a set of multiple knee X-ray images or a set of dynamic knee X-ray images and their corresponding mask label images to construct the knee X-ray image segmentation model; wherein, the mask label images include: patellar mask label; or, the mask label images include: one or more of femoral mask label, tibial mask label, patellar tendon mask label, and patellar mask label.

[0230] The construction of the knee joint X-ray image segmentation model includes: constructing a multi-segmentation region weight loss function based on the mask label images corresponding to a set of multiple knee joint X-ray images or a set of dynamic knee joint X-ray images; wherein, the mask label images include one or more of the following: patellar mask label, femoral mask label, tibial mask label, and patellar tendon mask label; using the set of multiple knee joint X-ray images or the set of dynamic knee joint X-ray images and their corresponding mask label images, and the multi-segmentation region weight loss function, multiple segmentation regions are segmented... The network is trained to obtain multiple knee joint X-ray image segmentation models or multiple dynamic knee joint X-ray image segmentation models; the multiple knee joint X-ray image segmentation models or the multiple dynamic knee joint X-ray image segmentation models are evaluated to determine the corresponding optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model; based on the optimal knee joint X-ray image segmentation model or optimal dynamic knee joint X-ray image segmentation model, the dynamic knee joint X-ray image to be segmented is segmented into one or more of the patellar tendon, femur, tibia and patella.

[0231] The step of determining the patella length of the knee X-ray image to be processed based on the coordinates of the third key point and the fourth key point includes: calculating the distance between the coordinates of the third key point and the fourth key point to determine the patella length of the knee X-ray image to be processed.

[0232] Constructing the keypoint detection model corresponding to the keypoint detection network includes: obtaining the third keypoint position coordinate labels corresponding to the highest point of the upper edge of the patella and the fourth keypoint position coordinate labels corresponding to the lowest point of the lower edge of the patella from multiple knee X-ray images; training the preset keypoint detection network using the first keypoint position labels and the second keypoint position labels to obtain the keypoint detection model.

[0233] Before training the preset keypoint detection network using the first keypoint location label and the second keypoint location label, the process includes: pre-training the preset keypoint detection network using multiple preset pose points corresponding to human pose images and / or multiple preset facial feature points corresponding to facial images to obtain a pre-trained keypoint detection model; then, re-training the pre-trained keypoint detection model using the first keypoint location label and the second keypoint location label to obtain a keypoint detection model.

[0234] The entity executing the image processing method corresponding to the determination of the contact surface and length of the femur and patella can be an image processing device or apparatus. For example, the image processing method corresponding to the determination of the contact surface and length of the femur and patella can be executed by a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the image processing method corresponding to the determination of the contact surface and length of the femur and patella can be implemented by a processor calling computer-readable instructions stored in memory.

[0235] Those skilled in the art will understand that in the above-described methods for determining the contact surface and length of the femur and patella and for image processing in specific embodiments, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined based on its function and possible internal logic.

[0236] According to one aspect of this disclosure, a knee joint X-ray imaging image processing system is provided, comprising: an acquisition unit for acquiring a femoral region mask image and a patellar region mask image corresponding to a knee joint X-ray image to be processed; a first determination unit for determining an overlapping region corresponding to the femoral region mask in the femoral region mask image and the patellar region mask image; a configuration unit for configuring the overlapping region as the contact surface between the femur and patella in the knee joint X-ray image to be processed; and / or The system includes: an acquisition unit for acquiring a femoral region mask image and a patellar region mask image corresponding to an X-ray image of the knee joint to be processed; a determination unit for determining the overlapping region corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; a configuration unit for configuring the overlapping region as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed; and a second determination unit for determining the length of the contact surface between the femur and patella based on the contact surface; and... An extraction unit is used to extract the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, a third determining unit is used to determine the first boundary line and the second boundary line among the boundary lines on both sides of the femur or femoral cortex; a fourth determining unit is used to determine the femoral midline based on the first boundary line and the second boundary line; and / or, using the key point detection model corresponding to the key point detection network, to detect the third key point position coordinates corresponding to the highest point coordinates of the upper edge of the patella and the fourth key point position coordinates corresponding to the lowest point coordinates of the lower edge of the patella in the knee X-ray image to be processed, and determine the third key point position coordinates and the fourth key point position coordinates; based on the third key point position coordinates and the fourth key point position coordinates, determine the patella length of the knee X-ray image to be processed.

[0237] According to one aspect of this disclosure, an X-ray camera is provided, including: the knee joint X-ray imaging image processing system as described above.

[0238] According to one aspect of this disclosure, an X-ray camera is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above-described method for determining the contact surface of the femur and patella; or, comprising: a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the above-described method for determining the contact surface of the femur and patella to generate the bit stream; or, comprising: a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the above-described method for determining the contact surface of the femur and patella.

[0239] According to one aspect of this disclosure, an X-ray machine is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the aforementioned method for determining the contact surface length of the femur and patella; or, comprising: a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the aforementioned method for determining the contact surface length of the femur and patella to generate the bit stream; or, comprising: a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the aforementioned method for determining the contact surface length of the femur and patella.

[0240] According to one aspect of this disclosure, an X-ray camera is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described knee joint X-ray image processing method; or, comprising: a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the above-described knee joint X-ray image processing method to generate the bit stream; or, comprising: a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the above-described knee joint X-ray image processing method.

[0241] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0242] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0243] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured as described above. The electronic device can be provided as a terminal, a server, or other type of device.

[0244] Electronic device 800 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. Electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0245] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0246] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0247] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0248] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0249] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0250] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0251] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0252] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0253] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0254] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0255] For example, electronic device 1900 can be provided as a server. Electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 1922 is configured to execute instructions to perform the methods described above.

[0256] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0257] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0258] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0259] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0260] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0261] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0262] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0263] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining the contact surface between the femur and patella, characterized in that, include: Obtain the femoral region mask image and patellar region mask image corresponding to the X-ray image of the knee joint to be processed; Determine the overlapping areas corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; The overlapping region is configured as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed.

2. The method for determining the contact surface between the femur and patella according to claim 1, characterized in that, Before obtaining the femoral region mask image and patellar region mask image corresponding to the knee joint X-ray image, the method includes: segmenting the femur and patella of the knee joint X-ray image to be processed using a knee joint X-ray image segmentation model to obtain the femoral region mask image and patellar region mask image corresponding to the knee joint X-ray image to be processed; and / or, Constructing the knee joint X-ray image segmentation model includes: training a segmentation network using a set of multiple knee joint X-ray images or a set of dynamic knee joint X-ray images and their corresponding mask label images to construct the knee joint X-ray image segmentation model; wherein the mask label images include: femoral mask labels and patellar mask labels; and / or, Constructing the knee joint X-ray image segmentation model includes: training multiple segmentation networks using the specified training set of multiple knee joint X-ray images or the specified dynamic knee joint X-ray image training set and their corresponding mask label images to obtain multiple knee joint X-ray image segmentation models; evaluating the multiple knee joint X-ray image segmentation models to determine the corresponding optimal knee joint X-ray image segmentation model; and / or, The step of determining the optimal knee X-ray image segmentation model includes: determining multiple first comprehensive evaluation scales corresponding to multiple evaluation scales of multiple knee X-ray image segmentation models or multiple dynamic knee X-ray image segmentation models on a set of knee X-ray images, where the values ​​of these scales are positively proportional or positively correlated with the performance of the segmentation models, and multiple second comprehensive evaluation scales where the values ​​of these scales are inversely proportional or negatively correlated with the performance of the segmentation models; determining first score vectors corresponding to the multiple knee X-ray image segmentation models under the multiple first comprehensive evaluation scales and second score vectors corresponding to the multiple dynamic knee X-ray image segmentation models under the multiple second comprehensive evaluation scales; and evaluating the multiple knee X-ray image segmentation models based on the first score vectors and the second score vectors to determine the optimal knee X-ray image segmentation model.

3. The method for determining the contact surface between the femur and patella according to claim 2, characterized in that, During the training of a set segmentation network or multiple set segmentation networks, the process includes: acquiring the masked image corresponding to the set dynamic knee X-ray image validation set and setting a maximum number of iterations; if the number of iterations of the set segmentation network is less than the maximum number of iterations, and the loss value between the set multiple knee X-ray images training set and its corresponding masked image is less than a set loss value, and the first average intersection-union ratio (IUU) between the set multiple knee X-ray images validation set and its corresponding masked image at the first set number of iterations is greater than a set first average IUU value, then the second average IUU value between the smallest masked segmentation region in the N masked segmentation regions corresponding to the masked image and the smallest masked segmentation region in the set dynamic knee X-ray image validation set is greater than a set second average IUU value, then the set segmentation network is controlled to stop training; and / or, The training process for the specified segmentation network or multiple specified segmentation networks further includes: if the number of iterations of the specified segmentation network reaches the specified maximum number of iterations, then controlling the specified segmentation network to stop training; and / or, the training process for the specified segmentation network or multiple specified segmentation networks further includes: if the number of iterations of the specified segmentation network or multiple specified segmentation networks is less than the specified maximum number of iterations and the loss value between the specified multiple knee X-ray images training set and its corresponding mask label image is less than a specified loss value and the loss value between the specified multiple knee X-ray images validation set and its corresponding mask label image... If the first average intersection-union ratio (IUU) corresponding to the first set number of iterations is greater than the first average IUU set value, and the second average IUU corresponding to the smallest area mask segmentation region among the N mask segmentation regions corresponding to the mask label image and the smallest area mask segmentation region in the training and validation sets of multiple knee X-ray images is greater than the second average IUU set value, then the third average IUU between the training and validation sets of multiple knee X-ray images and their corresponding mask label images is calculated within the second set number of iterations; if the third average IUU is less than or equal to the set fluctuation value, then the set segmentation network is controlled to stop training.

4. The method for determining the contact surface between the femur and patella according to any one of claims 1-3, characterized in that, Also includes: Based on the loss function weights of each of the N masked segmentation regions, a multi-segmentation region weight loss function is constructed; the predetermined segmentation network is trained using the predetermined training set of multiple knee X-ray images or the predetermined dynamic knee X-ray image training set and its corresponding masked label images, and the multi-segmentation region weight loss function; and / or, The step of constructing a multi-segmentation region weighted loss function based on the loss function weights of each masked segmentation region in the training set of multiple knee joint X-ray images includes: obtaining the base loss function corresponding to each masked segmentation region; multiplying the loss function weights of each masked segmentation region by their respective base loss functions and then summing the results to construct the multi-segmentation region weighted loss function; wherein the base loss function corresponding to the loss function weights is configured as a cross-entropy loss function or a weighted cross-entropy loss function; and / or, Determining the loss function weights for each masked segmentation region includes: determining the areas of N masked segmentation regions corresponding to the masked label images in the knee joint X-ray image training set; calculating N ratios between the area of ​​each of the N masked segmentation regions and the total area of ​​the masked segmentation regions corresponding to the N masked segmentation regions; and determining the loss function weights for each masked segmentation region corresponding to the defined segmentation network based on N different products corresponding to any N-1 masked segmentation region areas among the N masked segmentation regions.

5. The method for determining the contact surface between the femur and patella according to any one of claims 1-4, characterized in that, Also includes: If the training set of multiple knee X-ray images is configured as a dynamic knee X-ray image training set, then the similarity between the multiple pre-labeled knee X-ray images and the multiple knee X-ray images corresponding to the multiple pre-labeled knee X-ray images in each example of the dynamic knee X-ray image is calculated. If the similarity of the multiple images is less than or equal to a preset image similarity, then the knee X-ray image corresponding to the similarity of the images is determined as the dynamic knee X-ray image to be labeled. And / or, It also includes: if the initial preset-labeled X-ray image corresponding to the plurality of preset-labeled X-ray images is configured as the X-ray image corresponding to a non-initial time in each example of dynamic X-ray images, then the similarity between the initial preset-labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the plurality of times preceding the initial preset-labeled X-ray image is further calculated; and / or, It also includes: if the last preset-labeled X-ray image corresponding to the plurality of preset-labeled X-ray images is configured as the X-ray image corresponding to a non-last moment in each example of dynamic X-ray images, then the similarity between the last preset-labeled X-ray image in each example of dynamic X-ray images and the plurality of X-ray images corresponding to the times after the last preset-labeled X-ray image is also calculated.

6. A method for determining the contact surface length between the femur and patella, characterized in that, include: Using the method for determining the contact surface of the femur and patella as described in any one of claims 1-5, the contact surface of the femur and patella in the X-ray image of the knee joint to be processed is determined; Based on the contact surfaces, the contact surface lengths of the femur and patella are determined.

7. The method for determining the contact surface length of the femur and patella according to claim 6, characterized in that, Determining the contact surface length of the femur and patella based on the contact surface includes: calculating multiple non-zero coordinate distances between non-zero coordinates in the contact surface; determining the maximum distance among the multiple non-zero coordinate distances as the contact surface length of the femur and patella; and / or, The step of calculating the distances between multiple non-zero coordinates in the contact surface includes: constructing a vector array using the non-zero coordinates in the contact surface; and calculating the distances between the currently traversed non-zero coordinate and other non-zero coordinates by traversing each non-zero coordinate in the vector array.

8. A method for processing X-ray images of the knee joint, characterized in that, include: The method for determining the contact surface of the femur and patella as described in any one of claims 1-5; and / or, the method for determining the length of the contact surface of the femur and patella as described in any one of claims 6-7; and, Extract the femoral mask boundary image corresponding to the femoral mask from the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, determine one or both of the first and second boundary lines among the boundary lines on both sides of the femur or femoral cortex; and / or, take the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, determine the first and second boundary lines among the boundary lines on both sides of the femur or femoral cortex; based on the first and second boundary lines, determine the femoral midline. And / or, using the keypoint detection model corresponding to the keypoint detection network, the position coordinates of the third keypoint corresponding to the highest point of the upper edge of the patella and the position coordinates of the fourth keypoint corresponding to the lowest point of the lower edge of the patella in the knee X-ray image to be processed are detected, and the position coordinates of the third keypoint and the fourth keypoint are determined; based on the position coordinates of the third keypoint and the fourth keypoint, the patella length of the knee X-ray image to be processed is determined.

9. A knee joint X-ray imaging image processing system, characterized in that, include: The acquisition unit is used to acquire the femoral region mask image and the patellar region mask image corresponding to the X-ray image of the knee joint to be processed; The first determining unit is used to determine the overlapping area corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; A configuration unit is used to configure the overlapping region as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed; And / or, including: an acquisition unit, configured to acquire a femoral region mask image and a patellar region mask image corresponding to the X-ray image of the knee joint to be processed; a determination unit, configured to determine the overlapping region corresponding to the femoral region mask in the femoral region mask image and the patellar region mask in the patellar region mask image; a configuration unit, configured to configure the overlapping region as the contact surface between the femur and patella in the X-ray image of the knee joint to be processed; a second determination unit, configured to determine the length of the contact surface between the femur and patella based on the contact surface; and, An extraction unit is used to extract the femoral mask boundary image corresponding to the femoral mask in the femoral region mask image; based on the non-zero coordinates of the femoral mask boundary in the femoral mask boundary image, a third determining unit is used to determine the first boundary line and the second boundary line among the boundary lines on both sides of the femur or femoral cortex; a fourth determining unit is used to determine the femoral midline based on the first boundary line and the second boundary line; and / or, using the key point detection model corresponding to the key point detection network, to detect the third key point position coordinates corresponding to the highest point coordinates of the upper edge of the patella and the fourth key point position coordinates corresponding to the lowest point coordinates of the lower edge of the patella in the knee X-ray image to be processed, and determine the third key point position coordinates and the fourth key point position coordinates; based on the third key point position coordinates and the fourth key point position coordinates, determine the patella length of the knee X-ray image to be processed.

10. An X-ray camera, comprising: The knee joint X-ray imaging image processing system as described in claim 9; or, The method comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke instructions stored in the memory to execute the method for determining the contact surface of the femur and patella according to any one of claims 1-5; or, a computer-readable storage medium storing a computer program / instructions and a bit stream thereon, wherein the computer program / instructions, when executed by a processor, implement the method for determining the contact surface of the femur and patella according to any one of claims 1-5 to generate the bit stream; or, a computer program product configured with a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the method for determining the contact surface of the femur and patella according to any one of claims 1-5; or, The method comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke instructions stored in the memory to execute the method for determining the contact surface length of the femur and patella as described in any one of claims 6-7; or, a computer-readable storage medium having a computer program / instructions and a bit stream stored thereon, wherein the computer program / instructions, when executed by a processor, implement the method for determining the contact surface length of the femur and patella as described in any one of claims 6-7 to generate the bit stream; or, a computer program product having a computer program / instructions configured to implement the method for determining the contact surface length of the femur and patella as described in any one of claims 6-7 when executed by a processor; or, The method comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the knee X-ray image processing method of claim 8; or, a computer-readable storage medium having a computer program / instructions and a bit stream stored thereon, wherein the computer program / instructions, when executed by a processor, implement the knee X-ray image processing method of claim 8 to generate the bit stream; or, a computer program product having a computer program / instructions configured thereon, wherein the computer program / instructions, when executed by a processor, implement the knee X-ray image processing method of claim 8.

Images