Automatic planning method and system for knee joint simulation surgery based on deep learning

Abstract

The present invention provides a deep learning-based automated planning method and system for simulated knee joint surgery. The method comprises: acquiring a medical image of the knee joint and converting pixels in the image into actual lengths; selecting key area images of the knee joint image, performing target detection on the key area images, and extracting key points; using the coordinates of the extracted key points, annotating the key points and force lines, calculating the correction angle, and then calculating the rotation angle, simulating bone cutting and rotation to obtain a surgical planning image, and automatically planning two types of surgeries: high tibial osteotomy and distal femoral osteotomy. The present invention uses deep learning technology to predict the locations of key points and calculate the angles of force lines in lower limb X-rays. The surgeon can select from eight surgical options based on the displayed angles. The system automatically plans and simulates the options, eliminating the need for manual operation and significantly improving work efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method and system for automatic planning of knee joint simulation surgery based on deep learning. Background Art

[0002] Knee osteoarthritis is one of the most common orthopedic conditions. High tibial osteotomy and distal femoral osteotomy are commonly used in clinical surgery to correct abnormal lower limb force alignment and extend the lifespan of the knee joint. Preoperative diagnosis and planning require full-length X-rays of the patient's lower limbs. Identifying the key points for knee surgery requires a significant amount of time for specialized surgeons, and the sheer volume of surgical planning limits the number of patients they can treat per hour. Summary of the Invention

[0003] In view of this, the present invention designs an automatic planning method and system for knee joint simulation surgery based on deep learning. It uses deep learning technology to predict the position of key points and calculate the angle of force lines in lower limb X-rays for doctors to choose. It can effectively assist doctors in planning surgery around the knee joint without the need for manual operation, greatly improving work efficiency.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0005] The deep learning-based automatic planning method for knee joint simulation surgery includes:

[0006] Acquiring a medical image of a knee joint, and converting pixels in the medical image of the knee joint into actual lengths;

[0007] Select key area images of medical images of knee joints, perform target detection on the key area images, and extract key points;

[0008] By using the coordinates of the extracted key points and marking the key points and force lines, the correction angle is calculated, and then the rotation angle is calculated. The bone cutting and bone rotation are simulated to obtain the surgical planning image, and automatic planning of two types of surgeries, high tibial osteotomy and distal femoral osteotomy, is performed.

[0009] Furthermore, converting pixels in the medical image of the knee joint into actual length includes:

[0010] Obtain a ruler image, filter out white pixels in the ruler image, and then detect the ruler image using the LSD line detection algorithm;

[0011] After obtaining the coordinates of the first and last endpoints of the ruler, the coordinates of the first and last endpoints of the ruler (X A and X B) Convert the pixel difference in the X direction to the millimeter unit to obtain the relationship coefficient α between the millimeter unit and the coordinate value, where

[0012] Furthermore, the key area images of the medical image of the knee joint are selected, and target detection is performed on the key area images. The key points extracted include:

[0013] Select images of the hip joint, knee joint, and ankle joint as key area images;

[0014] The YOLOX neural network algorithm is used to detect targets in images of the hip, knee, and ankle joints.

[0015] The PFDNet network is used to extract the center point of the hip joint, the hinge point of the knee joint area, the Fujisawa point, the operation point, and the center point of the ankle joint as key points.

[0016] Furthermore, using the coordinates of the extracted key points, by marking the key points and force lines, the correction angle is calculated including:

[0017] The line connecting the hip joint center and the Fujisawa point and the extended line are set as the target force line, called line 1, and the line from the hinge point to the ankle joint center is called line 2. With the hinge point as the center, the length of line 2 is the radius and is rotated to line 1. At this time, the ankle joint center coincides with line 1, resulting in line 3. The angle between line 3 and line 2 is the planned correction angle of the high tibial osteotomy.

[0018] The line connecting the ankle joint center and the Fujisawa point and the extended line are set as the target force line, called line 1', and the line from the hinge point to the hip joint center is called line 2'. With the hinge point as the center of the circle and the length of line 2' as the radius, the circle is rotated to line 1'. At this time, the hip joint center coincides with line 1', and line 3' is obtained. The angle between line 3' and line 2' is the planned correction angle of the distal femoral osteotomy.

[0019] The present invention also provides a deep learning-based automatic planning system for knee joint simulation surgery, comprising

[0020] An image preprocessing unit, configured to acquire a medical image of the knee joint and convert pixels in the medical image of the knee joint into actual lengths;

[0021] A key point extraction unit is used to select a key area image of a medical image of a knee joint, perform target detection on the key area image, and extract key points;

[0022] The automatic planning unit is used to use the coordinates of the extracted key points, annotate the key points and force lines, calculate the correction angle, and then calculate the rotation angle, simulate bone cutting and bone rotation to obtain the surgical planning image, and perform automatic planning for two types of surgeries: high tibial osteotomy and distal femoral osteotomy.

[0023] Compared to existing technologies, the deep learning-based automatic planning method and system for knee joint simulation surgery described in the present invention have the following advantages: a method and system for planning surgery around the lower limb knee joint of the human body is designed through coordinate calculation and image processing technology, and a regional target detection algorithm and a key point detection algorithm are used to automatically process full-length X-rays of the lower limb to simulate eight surgical methods around the knee joint. Automatic extraction of key points can effectively shorten the time it takes for doctors to manually select points, and automatic simulation of surgical planning and display of results can effectively shorten the time it takes to plan surgery, thereby significantly improving the diagnosis time for each patient, greatly reducing reliance on professionals, alleviating the burden on doctors, and improving diagnosis and treatment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0025] Figure 1 Schematic diagram of the automatic planning method for knee joint simulation surgery based on deep learning of the present invention;

[0026] Figure 2 Schematic diagram of a ruler image of the present invention;

[0027] Figure 3 This is a schematic diagram of the detection results of the LSD algorithm of the present invention;

[0028] Figure 4 This is a schematic diagram of the original lower limb force line image of the present invention;

[0029] Figure 5 It is a schematic diagram of the prediction results of the present invention;

[0030] Figure 6 is a hip joint region diagram of the present invention;

[0031] Figure 7 is a regional diagram of the knee joint of the present invention;

[0032] Figure 8 is a diagram of the ankle joint region of the present invention;

[0033] Figure 9 This is a diagram for planning the medial-open high proximal tibial osteotomy of the present invention;

[0034] Figure 10 Schematic diagram of the straight line angle calculation method of the present invention;

[0035] Figure 11 A schematic diagram of the rotation of a rectangular coordinate system to a polar coordinate system image according to the present invention;

[0036] Figure 12 This is a rendering of the medial-open high proximal tibial osteotomy of the present invention;

[0037] Figure 13 This is a plan diagram for the lateral open distal femoral osteotomy of the present invention;

[0038] Figure 14 This is a rendering of the effect of the lateral open distal femoral osteotomy of the present invention;

[0039] Figure 15 This is the preoperative planning effect of various HTO and DFO surgeries of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0041] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0042] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0043] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0044] Explanation of professional terms

[0045] DFO: distal femoral osteotomy

[0046] HTO: High tibial osteotomy

[0047] like Figure 1 As shown, the present invention provides an automatic planning method for knee joint simulation surgery based on deep learning, including

[0048] Step 1: Acquire a medical image of a knee joint, and convert pixels in the medical image of the knee joint into actual lengths;

[0049] The criterion in the actual surgical process is the actual size, so it is necessary to convert the pixels in the image into actual length, that is, to obtain the pixel spacing. Figure 2 Shown is an image of a ruler, each ruler having eighteen intervals of ten millimeters.

[0050] According to the characteristics of the image, the pixel threshold is first preset, and then the white pixels in the image are filtered out. Then the ruler image is detected by the LSD (Line Segment Detector) line detection algorithm. The results are as follows Figure 3 After obtaining the coordinates of the first and last endpoints of the ruler, the coordinates of the two endpoints of the ruler (X A and X B ) in the X-direction is converted to millimeters to obtain the relationship coefficient α between millimeters and coordinate values, as shown in Formula (1). After conversion, subsequent processing yields length information, not pixel information. This allows surgical planning to be selected based on surgical planning criteria, as pixel information cannot be used as a criterion.

[0051]

[0052] Step 2: Select the key area image of the medical image of the knee joint, perform target detection on the key area image, and extract key points;

[0053] The present invention selects key areas containing key points and does not perform global processing on the entire image in order to reduce the size of the model and reduce noise. The present invention uses the deep learning YOLOX neural network algorithm to perform target detection on the hip joint, knee joint, and ankle joint areas. Figure 4-5 As shown, Figure 4 In the figure, the original lower limb X-ray image passed into YoloX is shown. Figure 5 This is the result of the YoloX network after region selection. Figure 6-8 For the selected key areas, take the left lower limb as an example.

[0054] In the key area map, the PFDNet network is used to extract the hip joint center point, the hinge point of the knee joint area, the Fujisawa point, the operation point, and the ankle joint center point. The extracted key points are as follows Figure 9Hip center is the center of the hip joint, Hinge is the hinge point, Surgery is the surgical site, and Ankle center is the center of the ankle joint.

[0055] Step 3: Using the coordinates of the extracted key points, by marking the key points and force lines, the correction angle is calculated, and then the rotation angle is calculated. The bone cutting and bone rotation are simulated to obtain the surgical planning image, and automatic planning of two types of surgeries, high tibial osteotomy and distal femoral osteotomy, is performed.

[0056] The obtained hip joint center coordinates, ankle joint center coordinates, hinge point coordinates, operation point coordinates and Fujisawa point coordinates are used to plan the two major types of HTO and DFO surgeries. Figure 9 As shown, for the four types of surgeries in HTO surgery, the correction force line selection and image rotation method are all cut from the surgical point, and the lower half of the image is rotated with the hinge point as the center. The present invention takes the medial opening proximal tibial osteotomy as an example. The line connecting the hip joint center and the Fujisawa point and the extended line are taken as the target force line, called line 1, and the line from the hinge point to the ankle joint center is line 2. With the hinge point as the center of the circle, the length of line 2 is the radius and rotated to line 1. At this time, the center of the ankle joint coincides with line 1 (target force line), and line 3 is obtained. The angle between line 3 and line 2 is the planned correction angle of the medial opening proximal tibial osteotomy.

[0057] Specifically, the linear equation of line 1 (target force line) determined by the hip joint center and the Fujisawa point is defined as shown in formula (2), where (x0, y0) is the coordinate of the hinge point, (x1, y1) is the coordinate of the parameter point that determines line 1, and (x2, y2) is the coordinate of the original ankle joint center point.

[0058] y1=k1x1+b1 (2)

[0059] The length of line 2 is determined by the Euclidean distance formula (3):

[0060]

[0061] As shown in formula (4), assuming that the coordinates of the center point of the ankle joint after rotation are (x3, y3), the distance from the hinge point (x0, y0) to the center point of the ankle joint after rotation is:

[0062]

[0063] As shown in formula (5), taking the linear equation of the target force line as the constraint condition, the coordinates of the center point of the ankle joint after rotation are obtained as follows:

[0064]

[0065] The angle calculation method in the present invention is as follows Figure 10 As shown in the figure, if ∠AHB is used to represent the angle at any position in the plane to be calculated, such as the angle formed by the Fujisawa point, hip center, and ankle center, the purpose of the surgery is to make the two sides of the angle coincide, that is, the force line of the lower limb coincides with the line connecting the hip center and the ankle center. In fact, the size of ∠AHB is the angle of rotation around the hinge point.

[0066] The calculation of ∠AHB is divided into two parts: angle 1 between line segment AH and the x-axis, and angle 2 between line segment BH and the y-axis. ∠AHB is as shown in formulas (6), (7), and (8).

[0067] ∠AHB=angle1+angle2(6)

[0068]

[0069]

[0070] In the surgical planning of the tibial end, whether it is closed or requires the lower half of the image to be rotated, the upper and lower parts of the image are divided according to the line connecting the hinge point and the surgical point, and the equation is shown in formula (9).

[0071] Y=k2X+b2 (9)

[0072] The rotation process of the origin is as follows Figure 11 As shown. The image is rotated using the polar coordinate modeling method. Point v is rotated around the origin by an angle of θ to obtain point v′. Assuming that the coordinates of point v are (u, w), then the coordinates of point v′ are (u′, w′). Let r be the distance from the origin to v, and φ be the angle between the vector from the origin to point v and the x-axis. Then v and v′ can be expressed as formulas (10) and (11), and their matrix form can be expressed as formula (12).

[0073]

[0074]

[0075]

[0076] Assuming that the hinge point coordinates are (x0, y0), when the image rotates around the hinge point, the hinge point is first moved to the coordinate origin. The movement process is shown in formula (13), where x and y are the distances from the hinge point (x0, y0) to the origin. After the hinge point (x0, y0) moves to the origin, it becomes (x′0, y′0).

[0077]

[0078] As shown in formula (14), the homogeneous coordinate representation of the translation and rotation matrices of the introduced image is transformed into a 3×3 form.

[0079]

[0080] Where t is the offset coefficient.

[0081] After the hinge point returns to the origin, it rotates θ degrees around the origin, as shown in formula (15).

[0082]

[0083] At this time, the hinge point is translated back to its original position, as shown in formula (16).

[0084]

[0085] Finally, the rotation matrix of the lower half of the image around the hinge point can be obtained as shown in formula (17).

[0086]

[0087] in accordance with Figure 8 In the figure, the correction angles of line 2 and line 3 are adjusted. The lower half of the image is rotated with the hinge point as the rotation center to obtain the image of the surgical planning. Figure 12 shown.

[0088] For DFO surgery, the present invention takes the lateral open distal femoral osteotomy as an example. First, the coordinates of the hip joint center, ankle joint center, hinge point, operation point and Fujisawa point are obtained through the above chapters. The line connecting the ankle joint center and the Fujisawa point and the extension line are used as the target force line. Figure 12 As shown in Figure 1, the target force line is called line 1', and the line from the hinge point to the center of the hip joint is called line 2'. With the hinge point as the center, line 2' is rotated to line 1' with the length of the circle as the radius. At this time, the center of the hip joint coincides with line 1', resulting in line 3'. The angle between line 3' and line 2' is the angle of planned correction. The formulas for solving the angle are shown in (6), (7), and (8).

[0089] As shown in formula (18), the equation of the line 1' (target force line) determined by the ankle joint center and the Fujisawa point is:

[0090] y5=k5x5+b5 (18)

[0091] In the surgical planning of the femoral end, the upper half of the image needs to be rotated. The upper and lower parts of the image are divided according to the line connecting the hinge point and the surgical point, and the equation is shown in formula (19).

[0092] y6=k6x6+b6 (19)

[0093] The solution method of line 2' and line 3' and the rotation method of the image are consistent with the medial open proximal tibial osteotomy. Figure 13 In the figure, the correction angles of line 2' and line 3' are calculated. The upper half of the image is rotated with the hinge point as the rotation center to obtain the preoperative planning image of the lateral open distal femoral osteotomy. Figure 14 shown.

[0094] The present invention realizes the preoperative planning of various types of DFO and HTO surgeries by using coordinate calculation and image processing technology. Figure 15 As shown, (a) is the preoperative planning diagram and effect diagram of medial closure DFO surgery, (b) medial opening DFO surgery, (c) lateral closure DFO surgery, (d) medial closure HTO surgery, (e) lateral closure HTO surgery, and (f) lateral opening HTO surgery.

[0095] The present invention adopts the method of segmenting key areas to reduce the scope of key point detection; the present invention uses image processing technology to convert the size information of the ruler into pixel spacing, thereby providing a reference for determining the surgical size; the present invention uses YOLOX and PFDNet networks to realize the selection of key areas and the detection of key points; the present invention realizes the automation of simulated surgical bone interception and bone rotation through coordinate calculation and image processing technology, and displays the final effect.

[0096] The present invention also provides a deep learning-based automatic planning system for knee joint simulation surgery, comprising

[0097] An image preprocessing unit, configured to acquire a medical image of the knee joint and convert pixels in the medical image of the knee joint into actual lengths;

[0098] A key point extraction unit is used to select a key area image of a medical image of a knee joint, perform target detection on the key area image, and extract key points;

[0099] The automatic planning unit is used to use the coordinates of the extracted key points, annotate the key points and force lines, calculate the correction angle, and then calculate the rotation angle, simulate bone cutting and bone rotation to obtain the surgical planning image, and perform automatic planning for two types of surgeries: high tibial osteotomy and distal femoral osteotomy.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automatic planning method for knee joint simulation surgery based on deep learning, characterized by: include: Acquiring a medical image of a knee joint, and converting pixels in the medical image of the knee joint into actual lengths, including: Obtain a ruler image, filter out white pixels in the ruler image, and then detect the ruler image using the LSD line detection algorithm; After obtaining the coordinates of the first and last endpoints of the ruler, the coordinates of the first and last endpoints of the ruler are X A and X B Convert the pixel difference in the X direction to the millimeter unit to obtain the relationship coefficient α between the millimeter unit and the coordinate value, where: Select key area images of the medical image of the knee joint, perform target detection on the key area images, and extract key points, including: Select images of the hip joint, knee joint, and ankle joint as key area images; The YOLOX neural network algorithm is used to detect targets in images of the hip, knee, and ankle joints. The PFDNet network is used to extract the center point of the hip joint, the hinge point of the knee joint area, the Fujisawa point, the surgical point, and the center point of the ankle joint as key points; The obtained coordinates of the hip joint center, ankle joint center, hinge point, surgical point and Fujisawa point are used to plan the two major types of surgeries: HTO and DFO. For the four types of HTO surgery, the correction force line and image rotation method are selected. The image is cut from the surgical point, and the lower half of the image is rotated with the hinge point as the center. The upper and lower parts of the image are divided according to the line connecting the hinge point and the surgical point. For DFO surgery, the upper half of the image needs to be rotated with the hinge point as the rotation center, and the upper and lower parts of the image are divided according to the line connecting the hinge point and the surgical point.

2. The deep learning-based automatic planning method for knee joint simulation surgery according to claim 1, characterized in that: Using the coordinates of the extracted key points, by marking the key points and force lines, the correction angle is calculated including: The line connecting the hip joint center and the Fujisawa point and the extended line are set as the target force line, called line 1, and the line from the hinge point to the ankle joint center is called line 2. With the hinge point as the center, the length of line 2 is the radius and is rotated to line 1. At this time, the ankle joint center coincides with line 1, resulting in line 3. The angle between line 3 and line 2 is the planned correction angle of the high tibial osteotomy. The line connecting the ankle joint center and the Fujisawa point and its extension line are set as the target force line, which is called line 1. ， , the hinge point to the center point of the hip joint is line 2 ， , with the hinge point as the center, line 2 ， The length of the radius is rotated to line 1 ， , at this time the center of the hip joint is aligned with line 1 ， Overlap and get line 3 ， , line 3 ， With line 2 ， The included angle is the planned correction angle of the distal femoral osteotomy.

3. Deep learning-based automatic planning system for knee joint simulation surgery, characterized by: include An image preprocessing unit is configured to obtain a medical image of a knee joint and convert pixels in the medical image of the knee joint into actual lengths, comprising: Obtain a ruler image, filter out white pixels in the ruler image, and then detect the ruler image using the LSD line detection algorithm; After obtaining the coordinates of the first and last endpoints of the ruler, the coordinates of the first and last endpoints of the ruler are X A and X B Convert the pixel difference in the X direction to the millimeter unit to obtain the relationship coefficient α between the millimeter unit and the coordinate value, where: The key point extraction unit is used to select the key area image of the medical image of the knee joint, perform target detection on the key area image, and extract key points, including: Select images of the hip joint, knee joint, and ankle joint as key area images; The YOLOX neural network algorithm is used to detect targets in images of the hip, knee, and ankle joints. The PFDNet network is used to extract the center point of the hip joint, the hinge point of the knee joint area, the Fujisawa point, the surgical point, and the center point of the ankle joint as key points; Automatic planning unit, used to plan HTO and DFO surgeries using the obtained hip joint center coordinates, ankle joint center coordinates, hinge point coordinates, surgical point coordinates, and Fujisawa point coordinates; For the four types of HTO surgery, the correction force line and image rotation method are selected. The image is cut from the surgical point, and the lower half of the image is rotated with the hinge point as the center. The upper and lower parts of the image are divided according to the line connecting the hinge point and the surgical point. For DFO surgery, the upper half of the image needs to be rotated with the hinge point as the rotation center, and the upper and lower parts of the image are divided according to the line connecting the hinge point and the surgical point.

Images