Program, information processing method, information processing device, and method for generating a learning model.

Abstract

To display an inner plate line indicating the maximum pelvic reduction point on a fluoroscopic image of a patient undergoing artificial hip joint replacement surgery. [Solution] The program acquires fluoroscopic images of the patient into a computer, inputs the acquired fluoroscopic images into a learning model to derive positional information on the patient's inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum, recognizes the inner plate line in the patient's pelvis in the acquired fluoroscopic images, recognizes acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when placing the cup in the patient's pelvis in the fluoroscopic images, derives the cup CE angle or bone coverage rate when assuming the placement of the cup in a state where the patient's pelvis has been shaved using a bone excavation reamer used in procedures for artificial hip joints, overlays the inner plate line and acetabular positional relationship information onto the fluoroscopic images, and executes a process to display the cup CE angle or bone coverage rate attached to the fluoroscopic images.

Description

Technical Field

[0001] The present invention relates to a program, an information processing method, an information processing apparatus, and a method for generating a learning model.

Background Art

[0002] A socket angle setting device for hip arthroplasty is known (for example, Patent Document 1). According to the socket angle setting device for hip arthroplasty described in Patent Document 1, a socket angle setting method based on the pelvic surface and the inter-teardrop line is provided.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the socket angle setting device for hip arthroplasty of Document 1, for a fluoroscopic image of a patient undergoing a procedure related to a hip joint, regarding the display of superimposing an inner plate line indicating the maximum cutting point when shaving the pelvis to place a cup included in the hip joint, it is not considered.

[0005] The present disclosure has been made in view of such circumstances, and an object thereof is to provide a program or the like that can display by superimposing an inner plate line indicating the maximum cutting point when shaving the pelvis to place a cup included in the hip joint on a fluoroscopic image of a patient undergoing a procedure related to the hip joint.

Means for Solving the Problems

[0006] Note: The patent number in Japanese Patent Application Laid-Open No. ********** is replaced with "**********" as it's not provided in the original text.In one approach, the program acquires fluoroscopic images of a patient undergoing artificial hip joint surgery, and when these images are input to a trained model, the program derives positional information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum by inputting the acquired fluoroscopic images. Based on the derived positional information for the inner plate, the program recognizes the inner plate line in the patient's pelvis in the acquired fluoroscopic images, indicating the point where the pelvis is shaved down to accommodate the cup included in the artificial hip joint. Based on the positional information of the anterior and posterior walls of the acetabulum obtained, the system recognizes acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image. The system derives the cup CE angle or bony coverage rate when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved using a bone excavation reamer used in the procedure for the artificial hip joint. The system then overlays the recognized inner plate line and the acetabular positional relationship information onto the fluoroscopic image and executes a process to display the derived cup CE angle or bony coverage rate attached to the fluoroscopic image.

[0007] One proposed method involves a computer acquiring fluoroscopic images of a patient undergoing artificial hip joint treatment, and inputting the acquired fluoroscopic images into a learning model trained to output positional information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum. The learning model then derives positional information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum from the patient, and based on the derived positional information of the inner plate, it recognizes the inner plate line in the patient's pelvis in the acquired fluoroscopic image, indicating the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint. Based on the derived positional information of the anterior and posterior walls of the acetabulum, the system recognizes acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image. The system then derives the cup CE angle or bony coverage rate when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved using a bone excavation reamer used in the procedure for the artificial hip joint. The system then overlays the recognized inner plate line and the acetabular positional relationship information onto the fluoroscopic image and executes a process to display the derived cup CE angle or bony coverage rate attached to the fluoroscopic image.

[0008] One proposed design includes an information processing device comprising: an acquisition unit that acquires fluoroscopic images of a patient undergoing artificial hip joint treatment; a position information derivation unit that derives position information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic images into a learning model that has been trained to output position information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum when an fluoroscopic image is input; and an inner plate line recognition unit that recognizes an inner plate line indicating the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint, based on the derived position information of the inner plate, in the patient's pelvis in the acquired fluoroscopic image. Based on the positional information of the anterior and posterior walls of the acetabulum, the device includes: an acetabular positional relationship information recognition unit that recognizes acetabular positional relationship information relating to the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image; a derivation unit that derives the cup CE angle or bony coverage rate when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved by a bone excavation reamer used in the procedure for the artificial hip joint; and a display unit that superimposes the recognized inner plate line and the acetabular positional relationship information onto the fluoroscopic image and displays the derived cup CE angle or bony coverage rate attached to the fluoroscopic image.

[0009] One proposed method for generating a learning model involves acquiring training data including fluoroscopic images of a patient undergoing artificial hip joint surgery and positional information of the patient's inner plate, and generating a learning model that outputs inner plate positional information when a fluoroscopic image is input, based on the acquired training data, wherein the fluoroscopic image includes multiple pseudo-fluoroscopic images generated at multiple different fluoroscopic angles from a DRR image converted from the patient's tomographic image. [Effects of the Invention]

[0010] According to this disclosure, it is possible to superimpose an inner plate line, which indicates the point of maximum reduction when the pelvis is shaved to position the cup included in the artificial hip joint, onto a fluoroscopic image of a patient undergoing artificial hip joint surgery. [Brief explanation of the drawing]

[0011] [Figure 1] It is a schematic diagram showing an overview of an intraoperative support system including an information processing apparatus according to Embodiment 1. [Figure 2] It is a block diagram showing a configuration example of an information processing apparatus. [Figure 3] It is an explanatory diagram showing an example of a learning model. [Figure 4] It is a flowchart showing an example of a processing procedure (during model learning) of a processing unit of an information processing apparatus. [Figure 5] It is an explanatory diagram showing an example of a second learning model (osseous coverage rate model). [Figure 6] It is a flowchart showing an example of a processing procedure (during model operation) of a processing unit of an information processing apparatus. [Figure 7] It is an explanatory diagram regarding the cup CE angle. [Figure 8] It is an explanatory diagram regarding the inner plate line (limit depth). [Figure 9] It is an explanatory diagram regarding acetabular position relationship information (anterior and posterior wall region information, central region information). [Figure 10] It is an explanatory diagram regarding the must-reach line. [Figure 11] It is an explanatory diagram regarding the upper limit line. [[ID=3s]] [Figure 12] It is an explanatory diagram regarding the reach area (strike zone). [Figure 13] It is an explanatory diagram regarding the inclination angle and abduction angle of the cup (assumed arrangement in a state where the pelvis is cut by a bone cutting reamer). [Figure 14] It is an explanatory diagram exemplifying a display screen of support information (support information superimposed screen). [Figure 15] It is an explanatory diagram exemplifying a display screen of support information (distortion correction screen).

Mode for Carrying Out the Invention

[0012] Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments. (Embodiment 1)

[0013] FIG. 1 is a schematic diagram showing an overview of an intraoperative support system S including an information processing apparatus 1 according to Embodiment 1. FIG. 2 is a block diagram showing a configuration example of the information processing apparatus 1. The intraoperative support system S is configured with the information processing apparatus 1 as a main apparatus. The information processing apparatus 1 is communicably connected to a fluoroscopic image capturing apparatus such as an X-ray apparatus 62 and a tomographic image capturing apparatus such as a CT apparatus 61 when generating a learning model 101 (position information model) or a second learning model 102 (bone coverage rate model). When outputting information regarding intraoperative support using the learning model 101 or the second learning model 102, the information processing apparatus 1 acquires in real time a fluoroscopic image (fluoroscopic image capturing) captured by the X-ray apparatus 62. When generating the learning model 101 or the second learning model 102, the information processing apparatus 1 acquires a CT image (tomographic image) captured by the CT apparatus 61. The information processing apparatus 1 may be further communicably connected to an electronic medical record server that stores and manages various medical data regarding the patient K.

[0014] For a patient K undergoing a procedure on the artificial hip joint 8, a fluoroscopic image (fluoroscopic image capturing) is captured by the X-ray apparatus 62 during the intraoperative period when the pelvis is being drilled by the bone drilling reamer 7. Therefore, the fluoroscopic image includes the bone drilling reamer 7 that drills the pelvis (during drilling). The X-ray apparatus 62 is configured to be rotatable relative to the patient K, and may move in the circumferential direction over, for example, 180 degrees or 360 degrees with the height direction (body length direction) of the patient K as the rotation axis (Z-axis). By rotating the X-ray apparatus 62 with respect to the body axis (Z-axis) of the patient K and capturing a fluoroscopic image (fluoroscopic image capturing) of the patient K, a plurality of fluoroscopic images (fluoroscopic image capturing) can be acquired at different fluoroscopic angles (X-ray irradiation angles) for the patient K. Therefore, the fluoroscopic image (fluoroscopic image capturing) includes a fluoroscopic image of the patient K's pelvis captured from the front and fluoroscopic images of the patient K's pelvis captured from the left and right side surfaces, respectively.

[0015] After the physician has performed appropriate drilling into the pelvis using a bone drilling reamer 7, the artificial hip joint 8, including the cup 81, is inserted (implanted) into the patient K's body. The artificial hip joint 8 includes a hemispherical cup 81 that fits into the drilled pelvis and a stem that is inserted into the femur. The bone drilling reamer 7 has a hemispherical tip for drilling, and the outer edge shape of this tip is the same shape and size as the outer edge shape of the cup 81 of the artificial hip joint 8.

[0016] As will be described in detail later, the processing unit 2 of the information processing device 1 recognizes the shape of the bone drilling reamer 7 (the hemispherical tip that performs drilling) included in the fluoroscopic image (X-ray image) captured and acquired during surgery, using an object detection model such as YOLO or edge detection, and identifies the position and inclination of the bone drilling reamer 7 in the pelvis. The processing unit 2 of the information processing device 1 considers the identified position and inclination of the bone drilling reamer 7 as the position and inclination of the cup 81 that is expected to be positioned when the pelvis is drilled by the bone drilling reamer 7, and calculates and outputs various information regarding the cup 81 that is expected to be positioned. Furthermore, the processing unit 2 of the information processing device 1 uses the inner plate line derived by the learning model 101 based on the fluoroscopic image (X-ray image) captured and acquired during or immediately before surgery to superimpose and display an inner plate line (reachable area) to support drilling by the bone drilling reamer 7 onto the fluoroscopic image (X-ray image) captured and acquired during surgery. Furthermore, the processing unit 2 of the information processing device 1 may use acetabular positional relationship information derived by the learning model 101 based on fluoroscopic images (X-ray images) acquired during or immediately before surgery to superimpose and display acetabular positional relationship information on the fluoroscopic images (X-ray images) acquired during surgery to support excavation by the bone excavation reamer 7. In this case, the acetabular positional relationship information may include anterior-posterior wall region information indicating the areas of the anterior wall (acetabular anterior wall) and posterior wall (acetabular posterior wall) of the acetabulum that the front and back of the cup 81 contact (or can contact) when the cup 81 is placed. This anterior-posterior wall region information allows for the recognition of the contact relationship between the anterior wall and posterior wall of the acetabulum and the cup 81 in the pelvis of patient K, so that the cup 81 can obtain sufficient initial fixation. Alternatively, the acetabular positional relationship information may include central region information indicating the region where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum. Similar to the anterior and posterior wall region information, this allows for the recognition of the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81, which is necessary for the cup 81 to achieve sufficient initial fixation.

[0017] The information processing device 1 is a computer capable of various information processing and information transmission and reception, such as a server device or a personal computer. The server device includes not only a single server device but also a cloud server device or virtual server device composed of multiple computers. If the information processing device 1 is configured as a cloud server device, for example, the information processing device 1 does not need to be installed in the medical facility where patient K is located, like medical equipment such as a CT scanner 61 or an X-ray scanner 62, and may be connected to these medical equipment via an external network such as the internet in a way that allows communication. The information processing device 1 includes a processing unit 2, a storage unit 3, an input / output interface 4, and a communication unit 5.

[0018] The processing unit 2 has one or more arithmetic processing units equipped with timing functions such as CPUs (Central Processing Units), MPUs (Micro-Processing Units), and GPUs (Graphics Processing Units), and performs various information processing, control processing, etc. related to the information processing unit 1 by reading and executing the program P (program product) stored in the storage unit 3.

[0019] The storage unit 3 includes volatile storage areas such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), and flash memory, as well as non-volatile storage areas such as EEPROM or hard disk. The storage unit 3 pre-stores programs P (program products) and data referenced during processing. The programs P (program products) stored in the storage unit 3 may be programs P (program products) read from a recording medium M that the information processing device 1 can read. Alternatively, programs P (program products) may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 3.

[0020] The memory unit 3 stores various medical data and values ​​determined in the preoperative plan for patient K. The memory unit 3 also stores the actual files that constitute the learning model 101 and the second learning model 102. These actual files may be configured as part of the program P (program product).

[0021] The communication unit 5 is a communication module or communication interface for communicating with an electronic medical record server or an information terminal such as a smartphone held by a medical professional, either by wire or wireless connection. Examples include a wired communication module such as an Ethernet® connector, a narrow-area wireless communication module such as WiFi® or Bluetooth®, or a wide-area wireless communication module such as 4G or 5G. The processing unit 2 communicates with the electronic medical record server or information terminal via the communication unit 5, for example, through a local network within the medical institution or an external network such as the Internet.

[0022] The input / output interface 4 is a communication interface compliant with a communication standard such as RS232C or USB. An input device such as a keyboard or a display device 41 such as an LCD is connected to the input / output interface 4. Furthermore, medical equipment such as a CT scanner 61 or an X-ray scanner 62 may also be connected to the input / output interface 4.

[0023] Figure 3 is an explanatory diagram showing an example of the learning model 101. The learning model 101 is a neural network (NN) that performs object detection, semantic segmentation, or instance segmentation, and is composed of, for example, RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, or SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), etc.

[0024] The learning model 101 outputs positional information of the inner plate and acetabulum in the pelvis, based on the input fluoroscopic image (X-ray image). The positional information of the inner plate may be indicated, for example, by a linear inner plate line indicating the periphery of the inner plate in the pelvis, or by a plurality of points constituting the inner plate line (a point cloud consisting of a plurality of inner plate corresponding points). The inner plate line is a line (boundary line) that displays the limit depth when excavating the pelvis, and the learning model 101 functions as a limit depth extraction model that extracts the limit depth contained in the input fluoroscopic image (X-ray image). The positional information of the acetabulum may be indicated, for example, by the regions of the anterior wall (anterior acetabular wall) and posterior wall (posterior acetabular wall) located anteriorly and posteriorly in the pelvis. Furthermore, the positional information of the acetabulum may be indicated by the region where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum. Furthermore, the positional information of the acetabulum may include the region of the load-bearing surface located between the anterior and posterior walls. Thus, the learning model 101 functions as a position information model that outputs position information of the medial plate and acetabulum in the pelvis. As described above, a fluoroscopic image (X-ray image) includes multiple fluoroscopic images with different fluoroscopic angles, and the learning model 101 may output position information of the medial plate (medial plate line), position information of the acetabulum (anterior and posterior wall regions, central region of cup 81), or position information of both (position information of the medial plate and acetabulum) depending on the fluoroscopic angle of the input fluoroscopic image. For example, if the fluoroscopic angle is set to image from the front of the pelvis, the learning model 101 may output position information of the medial plate (medial plate line), and if the fluoroscopic angle is set to image from the side of the pelvis, it may output position information of the acetabulum (anterior and posterior wall regions, central region of cup 81).

[0025] If the learning model 101 (location information model) is composed of a neural network including a CNN (Convolutional Neural Network) that extracts image features, such as RCNN, the input layer included in the learning model 101 has multiple neurons that accept image pixel values ​​as input and passes the input pixel values ​​to the hidden layer. The hidden layer has multiple neurons that extract image features and passes the extracted image features to the output layer. The output layer has one or more neurons that output region information including the positions of inner platen lines (point clouds corresponding to the inner platen), and outputs the positions of the inner platen lines (point clouds corresponding to the inner platen) (region coordinates or pixel numbers, etc.) based on the image features output from the hidden layer. Furthermore, the output layer has one or more neurons that output region information including the positions of the anterior and posterior walls of the acetabulum, and outputs the positions of the anterior and posterior walls of the acetabulum (region coordinates or pixel numbers, etc.) based on the image features output from the hidden layer. Alternatively, the output layer may have one or more neurons that output region information, including the position of the center of the cup 81, when the cup 81 is positioned in contact with the anterior and posterior walls of the acetabulum, and output the position of the center of the cup 81 (region coordinates or pixel number, etc.) based on the image features output from the intermediate layer. The center of the cup 81 is not limited to the center of the cup 81 itself, but may include the region that defines the center of the cup 81.

[0026] The neural network (learning model 101) trained using training data is intended to be used as a program module, which is part of artificial intelligence software. The learning model 101 is used in an information processing device 1, which has a processing unit 2 (CPU, etc.) and a memory unit 3, as described above. By being executed in the information processing device 1, which has such computational processing capabilities, a neural network system is formed. Specifically, the processing unit 2 of the information processing device 1 performs calculations to extract feature quantities from the image input to the input layer according to commands from the learning model 101 stored in the memory unit 3, and outputs the position (region) of the inner plate line (inner plate corresponding point cloud) from the output layer.

[0027] The learning model 101 can be generated by preparing training data in which problem data consisting of fluoroscopic images (X-ray images) including the pelvis (hip joint) are associated with labels (answer data) that indicate the position (region) of the medial plate line in the pelvis or the anterior and posterior walls of the acetabulum, and then using this training data to machine-learn an untrained neural network. In this case, the fluoroscopic images (X-ray images) may be pseudo-fluoroscopic images (DRR-derived fluoroscopic images [DRR-F]) generated using DRR images (Digital Reconstructed Radiograph: CT reconstruction simulation images) converted from tomographic images such as CT images including the pelvis (hip joint).

[0028] Training data is stored, for example, in the memory unit 3 of the information processing device 1 and can be generated by aggregating images and physician's findings contained in a large amount of diagnostic or surgical results stored on an electronic medical record server in a medical institution such as a hospital. In other words, the medial plate line in the pelvis (medial plate line, or the point group corresponding to the medial plate line that constitutes the medial plate line) and the anterior and posterior walls of the acetabulum are locations (internal body parts) identified based on findings by physicians, etc. According to the learning model 101 that has been learned and configured in this way, by inputting a fluoroscopic image (X-ray image) into the learning model 101, information indicating the position of the medial plate line of the pelvis (region coordinates in the image coordinate system) and the anterior and posterior walls of the acetabulum contained in the fluoroscopic image can be obtained. The learning model 101 may also output a fluoroscopic image in which the derived medial plate line and the anterior and posterior walls of the acetabulum are superimposed on the input fluoroscopic image (X-ray image).

[0029] In this embodiment, the learning model 101 is described as being RCNN, but the learning model 101 is not limited to RCNN, and may be a learning model 101 constructed with other learning algorithms such as neural networks other than RCNN, SVM (Support Vector Machine), transformers, YOLO, Bayesian networks, regression trees, etc. The dataset of problem data and answer data included in the training data for learning the learning model 101 and the dataset of input data and output data when using the learning model 101 are synonymous, and if defined in one dataset, it will naturally apply to the other dataset as well.

[0030] Figure 4 is a flowchart showing an example of the processing procedure (during model learning) of the processing unit 2 of the information processing device 1. The processing unit 2 of the information processing device 1 receives operator input, for example, from a keyboard connected to the input / output I / F 4, and performs the following processing based on the received operation.

[0031] The processing unit 2 of the information processing device 1 acquires tomographic images (S11). In generating the learning model 101, the processing unit 2 of the information processing device 1 acquires tomographic images (CT images) of the pelvises of a large number of subjects. These tomographic images (CT images) are taken on a subject or patient K using a CT scanner 61 and consist of multiple images (group of tomographic images) showing cross-sections of the body with a predetermined slice width. These tomographic images (CT images) are stored in an electronic medical record server or the like at a medical institution such as a hospital. The processing unit 2 of the information processing device 1 can acquire a large number of tomographic images (CT images) by accessing the electronic medical record server or the like.

[0032] The processing unit 2 of the information processing device 1 converts the acquired tomographic image into a DRR image (S12). The processing unit 2 of the information processing device 1 converts the tomographic image into a DRR image (Digital Reconstructed Radiograph: CT reconstruction simulation image), associates the converted DRR image with the original tomographic image data (a group of tomographic images showing cross-sections of the body at a predetermined slice width), and stores it in the storage unit 3. The processing unit 2 of the information processing device 1 may, for example, use raycasting processing, where the line of sight is replaced with an X-ray source, to convert the tomographic image into a DRR image.

[0033] The processing unit 2 of the information processing device 1 generates a pseudo-fluoroscopic image using the DRR image (S13). When generating a pseudo-fluoroscopic image using the DRR image, the processing unit 2 of the information processing device 1 may generate multiple pseudo-fluoroscopic images (DRR-F) with different fluoroscopic angles, i.e., different X-ray irradiation angles for the subject. This makes it possible to generate multiple pseudo-fluoroscopic images with different fluoroscopic angles from a single DRR image, and to efficiently generate multiple times the number of pseudo-fluoroscopic images compared to the original tomographic image (CT image), i.e., to perform training data augmentation (data argumentation).

[0034] The processing unit 2 of the information processing device 1 may perform angle correction on the original data, which is a tomographic image (CT image) or DRR image, when generating a pseudo-fluoroscopic image. The angle correction is performed so that the pseudo-fluoroscopic image (DRR-F) shows a complete frontal view of the pelvis, and the processing unit 2 of the information processing device 1 may modify the tomographic image (CT image) or DRR image based on the correction value of the angle correction.

[0035] The processing unit 2 of the information processing device 1 generates training data using a pseudo-transparency image (S14). The processing unit 2 of the information processing device 1 generates training data by adding (annotating) positional information of the pelvic inner plate and acetabulum contained in the pseudo-transparency image to the pseudo-transparency image. The positional information of the pelvic inner plate may be indicated, for example, by a linear inner plate line indicating the periphery of the pelvic inner plate, or by a plurality of points (inner plate corresponding points) that constitute the inner plate line. The positional information of the pelvic acetabulum may be indicated, for example, by an elliptical object indicating the anterior and posterior wall regions of the pelvis. Alternatively, the positional information of the pelvic acetabulum may be indicated, for example, by a circular object indicating the center of the cup 81 located inside the acetabulum of the pelvis. The positional information of the inner plate (inner plate line) and the positional information of the acetabulum (anterior-posterior wall region information, central region information) assigned to the pseudo-fluoroscopic image are determined in accordance with the locations (internal body parts) identified by physicians or other medical professionals based on their findings in the tomographic image (CT image), which is the source data for the pseudo-fluoroscopic image.

[0036] The processing unit 2 of the information processing device 1 may, when generating training data, annotate the positional information of the inner plate and acetabulum in the pelvis to the pseudo-fluoroscopic image based on tomographic images (CT images) on which the inner plate correspondence points (landmarks) and the regions of the anterior and posterior walls of the acetabulum have been marked by a physician or other medical professional. In this case, the processing unit 2 of the information processing device 1 can align the DRR image converted from the tomographic image (CT image) and the pseudo-fluoroscopic image (DRR-F) generated from the DRR image by applying the same internal coordinate system. Therefore, the processing unit 2 of the information processing device 1 may, based on the internal coordinate system, expand the inner plate correspondence points (landmarks) and the regions of the anterior and posterior walls, which have been marked on each of the multiple tomographic images with a predetermined slice width, onto the pseudo-fluoroscopic image, thereby annotating the positional information of the inner plate (inner plate line) and the positional information of the acetabulum in the pseudo-fluoroscopic image.

[0037] The processing unit 2 of the information processing device 1 generates a learning model 101 using the acquired training data (S15). The processing unit 2 of the information processing device 1 generates a learning model 101 by training an untrained neural network by applying the training data (images on which the positional information of the inner plate and acetabulum is annotated to pseudo-transspective images).

[0038] In this embodiment, the learning model 101 outputs positional information of the inner plate of the pelvis (inner plate line) and positional information of the acetabulum included in the input pseudo-fluoroscopic image, but is not limited to this. The learning model 101 may output a cup position derived based on the positional information of the inner plate of the pelvis (inner plate line) and positional information of the acetabulum included in the input pseudo-fluoroscopic image. In this case, the learning model 101 may output positional information (outer edge diagram of the cup 81 at the installation position when the cup 81 is installed) indicating a cup position that has a target line and a cup CE angle or bony coverage rate based on the input pseudo-fluoroscopic image. That is, the learning model 101 may output positional information (outer edge diagram of the cup 81 at the installation position when the cup 81 is installed) indicating the cup position when the pseudo-fluoroscopic image, the target line or bony coverage rate calculated based on the pseudo-fluoroscopic image, and the radius of the cup 81 and target CE angle determined in the preoperative plan are input. In this case, the learning model 101 may output a fluoroscopic image in which the derived cup position (outer edge diagram of cup 81) is superimposed on the input fluoroscopic image (X-ray image). In this case, the learning model 101 is trained to output the cup position based on these input data (pseudo-fluoroscopic image, required line, bone coverage rate, radius of cup 81, target CE angle).

[0039] Figure 5 is an explanatory diagram showing an example of the second learning model 102 (bone coverage model). The second learning model 102 is constructed, for example, using a CNN (Convolutional Neural Network). Based on the input fluoroscopic image (X-ray image), the second learning model 102 outputs the bone coverage ratio of the pelvis contained in the fluoroscopic image. The bone coverage ratio includes, for example, BCI (Bone Coverage Index) or three-dimensional bone coverage ratio. BCI is defined, for example, as the ratio of the horizontal width of the covered portion of cup 81 to the horizontal width of the entire cup 81. Three-dimensional bone coverage ratio is a ratio that shows in three dimensions how much of the femoral head is covered by the acetabulum (femoral head coverage state by the acetabulum). Thus, the second learning model 102 functions as a bone coverage ratio model that outputs the bone coverage ratio (BCI, three-dimensional bone coverage ratio) of the pelvis.

[0040] The second learning model 102, like the learning model 101, is trained on multiple perspective images with different perspective angles and consists of a neural network including a CNN (Convolutional Neural Network) that extracts image features. The input layer in the second learning model 102 has multiple neurons that accept image pixel values ​​as input and passes the input pixel values ​​to the hidden layer. The hidden layer has multiple neurons that extract image features from the image and passes the extracted image features to the output layer. The output layer has one or more neurons that output bone coverage and outputs bone coverage based on the image features output from the hidden layer.

[0041] The second learning model 102 can be generated by preparing training data in which problem data consisting of fluoroscopic images (X-ray images) including the pelvis (hip joint) are associated with labels (answer data) that indicate the bone coverage rate in the pelvis, and then using this training data to machine-learn an untrained neural network. The annotation of the bone coverage rate is performed based on the findings of a physician, etc., as with learning model 101. In this case, as with learning model 101, the fluoroscopic images (X-ray images) may also be pseudo-fluoroscopic images (DRR-derived fluoroscopic images [DRR-F]) generated using DRR images (Digital Reconstructed Radiograph: CT reconstruction simulation images) converted from tomographic images such as CT images including the pelvis (hip joint).

[0042] Figure 6 is a flowchart showing an example of the processing procedure (during model operation) of the processing unit 2 of the information processing device 1. The processing unit 2 of the information processing device 1 starts the processing shown in the flowchart, for example, when a perspective image is input as a trigger, or when it receives a start command from an input device such as a keyboard connected to the input / output I / F 4.

[0043] The processing unit 2 of the information processing device 1 obtains various values ​​determined in the preoperative plan by referring to the storage unit 3 (S101). The various values ​​determined in the preoperative plan include, for example, the radius of the cup 81 of the artificial hip joint 8 to be placed in the body, the target CE angle, the target bony coverage rate, the target distance between the outer edge and the inner plate line of the cup 81 which has been set in advance as a target value, the target anterior tilt angle, and the target abduction angle.

[0044] The target CE angle indicates the cup CE angle when the cup 81, which is set in advance as the target value, is positioned. The target distance indicates the distance between the cup 81, which is set in advance as the target value, and the inner plate line derived using the learning model 101 based on the tomographic image. The target anteversion angle indicates the anteversion angle of the cup 81 when the cup 81, which is set in advance as the target value, is positioned. The target abduction angle indicates the abduction angle of the cup 81 when the cup 81, which is set in advance as the target value, is positioned. These various values ​​determined in the preoperative plan are determined by a physician or the like, input in advance into the information processing device 1, and stored in the storage unit 3. Alternatively, the processing unit 2 of the information processing device 1 may acquire the various values ​​determined in the preoperative plan for patient K from the electronic medical record system based on the patient ID, etc.

[0045] The processing unit 2 of the information processing device 1 acquires a fluoroscopic image (X-ray image) of patient K's pelvis during surgery on patient K who is receiving treatment for the artificial hip joint 8 (S102). The processing unit 2 of the information processing device 1 acquires a fluoroscopic image (X-ray image) of patient K's pelvis in real time during surgery on patient K who is receiving treatment for the artificial hip joint 8, that is, while the pelvis is being excavated by the bone excavation reamer 7. The processing unit 2 of the information processing device 1 may acquire the fluoroscopic image (X-ray image) in real time in video format.

[0046] The fluoroscopic images (X-ray images) acquired in video format include fluoroscopic images (X-ray images) of patient K's pelvis before it is drilled by the bone drilling reamer 7, and fluoroscopic images (X-ray images) that include the bone drilling reamer 7. Thus, the fluoroscopic images (X-ray images) of patient K acquired in real time during surgery include the bone drilling reamer 7 that drills the pelvis, and the current position and inclination of the bone drilling reamer 7 (the outer edge of the hemispherical tip that drills) in the pelvis can be identified. Assuming the placement of the cup 81 in the bone cavity (installation space) drilled by the bone drilling reamer 7, the cup CE angle, anterior tilt angle, and abduction angle of the assumed-placed cup 81 can be derived.

[0047] Figure 7 is an explanatory diagram regarding the cup CE angle. In the illustration of this embodiment, a semicircular geometric object representing the assumed placement of the cup 81 based on the current position of the bone-excavating reamer 7 in the pelvis is superimposed on the perspective image. The processing unit 2 of the information processing device 1 derives the cup CE angle as the angle formed by the line connecting the intersection of the outer edge of the cup 81 and the white line (Acetabular sourcil) on the pelvic load surface, from the center (COR: Center of Rotation) of the cup 81 assumed to be placed in the state in which the pelvis has been excavated by the bone-excavating reamer 7, and the line connecting the lower ends of the left and right teardrop marks in the pelvis, which is perpendicular to the pelvic reference line (Y axis).

[0048] The processing unit 2 of the information processing device 1 uses the learning model 101 to derive the positional information (acetabular positional relationship information) of the inner plate line and the anterior and posterior walls of the acetabulum in the fluoroscopic image (S103). The processing unit 2 of the information processing device 1 may, for example, input a fluoroscopic image (X-ray image) that was captured and acquired immediately before the surgery, that is, a fluoroscopic image (X-ray image) of patient K's pelvis before it was excavated by the bone excavation reamer 7, into the learning model 101.

[0049] The learning model 101 outputs a fluoroscopic image in which the position information of the inner plate (inner plate line) and the position information of the acetabulum (acetabular positional relationship information) are superimposed on the input fluoroscopic image (X-ray image). The inner plate line indicates the maximum cutting point when cutting the pelvis, i.e., the drilling limit line (limit depth) that must not be exceeded. The position information of the acetabulum (acetabular positional relationship information) includes anterior-posterior wall region information indicating the regions of the anterior wall (acetabular anterior wall) and posterior wall (acetabular posterior wall) of the acetabulum, or central region information indicating the region where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum. This information indicates the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81, so that the cup 81 can obtain sufficient initial fixation. The processing unit 2 of the information processing device 1 derives (identifies) the inner plate line of the pelvis and the regions of the anterior and posterior walls of the acetabulum in the fluoroscopic image based on the output result (inner plate position information) from the learning model 101.

[0050] Figure 8 is an explanatory diagram relating to the inner plate line (critical depth). In the illustration of this embodiment, multiple (three in this embodiment) tomographic images are shown, and each tomographic image (CT image) has a correspondence relationship (same Y coordinate) with the fluoroscopic image (X-ray image) at multiple locations in the Y-axis direction (vertical direction of the human body). Each of these tomographic images was used when generating the learning model 101, and circles indicating inner plate corresponding points (landmarks) are superimposed on each tomographic image (CT image). That is, the pseudo-fluoroscopic image created using the DRR image converted from the tomographic image is given an inner plate line formed by connecting the inner plate corresponding points (landmarks) shown in these multiple tomographic images, and is used as training data when generating the learning model 101.

[0051] The processing unit 2 of the information processing device 1 derives the required line in the fluoroscopic image according to the radius of the cup 81 and the target CE angle (S104). Alternatively, the processing unit 2 of the information processing device 1 may derive the required line in the fluoroscopic image according to the radius of the cup 81 and a predetermined target bony coverage ratio for stabilizing the positioned cup 81. The radius of the cup 81 of the artificial hip joint 8 to be inserted into patient K and the target CE angle or target bony coverage ratio are stored in the storage unit 3 as various values ​​determined in the preoperative plan. The processing unit 2 of the information processing device 1 derives the required line in the fluoroscopic image based on the white line (Acetabular sourcil) formed by the inner edge of the pelvic load surface included in the fluoroscopic image, according to the radius of the cup 81 and the target CE angle or target bony coverage ratio. The white line (Acetabular sourcil) formed by the inner edge of the pelvic load-bearing surface can be identified (its position determined) by performing shape recognition processing on the fluoroscopic image. The required line indicates the minimum cutting point when trimming the pelvis in patient K to position the cup 81.

[0052] Figure 9 is an explanatory diagram relating to acetabular positional relationship information (anterior-posterior wall region information, central region information). In the illustration of this embodiment, annotations for generating the learning model 101, such as the regions of the anterior wall (acetabular anterior wall) and posterior wall (acetabular posterior wall), are added to an X-ray image (fluoroscopic image acquisition) taken at a fluoroscopic angle from the side of the pelvis, and these are used as training data when generating the learning model 101. In the X-ray image (fluoroscopic image acquisition) taken from the side of the pelvis, the acetabulum is shown as a circle, and the anterior and posterior walls of the acetabulum are located in front of and behind the acetabulum. When the cup 81 is placed, the front and back of the cup 81 can come into contact with the anterior and posterior walls of the acetabulum, and the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81 is recognized by the regions of the posterior and anterior walls (anterior-posterior wall region information). Furthermore, when the cup 81 is positioned, the region on the inside of the acetabulum where the center of the cup 81 may be located is shown as the central region of the cup 81 (central region information). Between the posterior wall region and the anterior wall region lies the load-bearing surface region. The learning model 101 may output the input X-ray image (fluoroscopic image acquisition) with the posterior and anterior wall regions (anterior-posterior wall region information), or the central region of the cup 81 (central region information) superimposed. Furthermore, the learning model 101 may also output the load-bearing surface region with the input X-ray image (fluoroscopic image acquisition) superimposed.

[0053] Furthermore, the processing unit 2 of the information processing device 1 may, for example, input a fluoroscopic image (X-ray image) captured and acquired immediately before the surgery, i.e., a fluoroscopic image (X-ray image) of patient K's pelvis before it was excavated by the bone excavation reamer 7, into the second learning model 102. The second learning model 102 outputs a bone coverage ratio, including BCI or three-dimensional bone coverage ratio, for the input fluoroscopic image (X-ray image). The processing unit 2 of the information processing device 1 may derive the bone coverage ratio (BCI or three-dimensional bone coverage ratio) by inputting multiple fluoroscopic images (X-ray images) taken at different fluoroscopic angles into the second learning model 102.

[0054] Figure 10 is an explanatory diagram regarding the required line. The storage unit 3 of the information processing device 1 stores various values ​​(parameters) determined in the preoperative plan when performing a procedure on the artificial hip joint 8, such as the minimum CE angle (e.g., 10 degrees) set in advance to stabilize the cup 81 to be placed, and the radius of the cup 81 to be implanted. The processing unit 2 of the information processing device 1 obtains these various values ​​determined in the preoperative plan by referring to the storage unit 3.

[0055] The processing unit 2 of the information processing device 1 defines the center (x,y) of the cup 81, which is assumed to be placed in the bone cavity (installation space) excavated by the bone excavation reamer 7, and the intersection point (X,Y) of the white line (Acetabular sourcil) representing the pelvic load surface at the time of excavation and the edge of the cup. In this case, the center (x,y) of the cup 81 is given by the following formula using the intersection point (X,Y), the radius (r) of the cup 81, and the cup CE angle (Θ). The X coordinate (x) of the center of the cup 81 is calculated by subtracting the value obtained by multiplying the radius (r) of the cup 81 by the sine of the cup CE angle (Θ) from the X coordinate (X) of the intersection point (x) (x = Xr * sinΘ). The Y-coordinate (y) of the center of cup 81 is calculated by subtracting the value obtained by multiplying the radius (r) of cup 81 by the cosine of the cup CE angle (Θ) from the Y-coordinate (Y) of the intersection point (y = Yr * cosΘ). When an arbitrary Θ is set, the center (x,y) of cup 81 corresponds one-to-one with the intersection point (X,Y). When the intersection point (X,Y) is changed, that is, when the amount of excavation by the bone excavation reamer 7 changes, the white line (Acetabular sourcil) indicating the pelvic load surface at the time of excavation also changes, the center (x,y) of cup 81 also changes. In this way, when varying the intersection point (X,Y), for example, the cup CE angle (Θ) must exceed the minimum CE angle (10 degrees) (Θ>10), which is a requirement for the implant. The processing unit 2 of the information processing device 1 may determine the range in which the intersection point (X,Y) can be varied within the range that satisfies this requirement (Θ>10). In other words, the numerical value of Θ is not a fixed value (correct value), and the cup CE angle (Θ) that ensures cup fixation may be displayed on the monitor (display device 41) and used as one of the factors in determining the depth to be mined.

[0056] The processing unit 2 of the information processing device 1 identifies the region in which the center (x,y) of the cup 81 can be located, based on the range of variation of the determined intersection point (X,Y), using the above formula. The processing unit 2 of the information processing device 1 takes into account the radius (r) of the cup 81 and the center (x,y) of the cup 81 determined according to the varied intersection point (X,Y), and derives the marginal lines in the range that each point (x',y') representing the outer edge of the cup 81 can take, as an example, a line that the outer edge of the bone excavation reamer 7 (the outer edge of the hemispherical tip that performs excavation) must touch in order to satisfy the requirement (Θ>10) (a must-reach line). In deriving this must-reach line, the processing unit 2 of the information processing device 1 selects an arbitrary point (x1,y1) on the acetabular base of the pelvis before excavation with the bone excavation reamer 7, and sets the angle of the X-axis with respect to the line connecting the center (x,y) of the cup 81 and the arbitrary point as α. The coordinates of any point are (x+rcosα, y+rsinα). Then, the processing unit 2 of the information processing device 1 may derive the required line such that the point (x+rcosα, y+rsinα), which is the center (x,y) of the cup 81 corresponding to the changed intersection point (X,Y) and takes into account the radius (r) of the cup 81, is greater than the coordinates (x1,y1) of the arbitrary point (x+rcosα>x1 and y+rsinα>y2). In other words, the processing unit 2 of the information processing device 1 derives the required line as the edge of the outer edge of the cup 81 by taking into account the range of (x',y') that the center (x,y) of the cup 81 satisfying the conditions "(x+rcosα>x1 and y+rsinα>y2)" and, as an example, "cup CE angle; Θ>10", i.e., the radius (r) of the cup 81. The resulting target line corresponds to the inner line within the reachable area (the inner side that is closer to the user in the drilling direction of the bone excavation reamer 7).

[0057] The processing unit 2 of the information processing device 1 derives an upward limit line in the fluoroscopic image according to the center of the femoral head on the healthy side (S105). The processing unit 2 of the information processing device 1 recognizes the femoral head on the healthy side included in the fluoroscopic image and derives an upward limit line in the fluoroscopic image according to the center of the femoral head on the healthy side. The upward limit line indicates the limit of upward movement that the center of the femoral head on the affected side (the center of the femoral head ball of the artificial hip joint 8) can take relative to the center of the femoral head on the healthy side.

[0058] Figure 11 is an explanatory diagram relating to the upward limit line. In the illustration of this embodiment, four diagrams ((1) to (4)) are shown for reference. These diagrams are as follows: Diagram (1) shows a line parallel to the X-axis passing through y'=y2+10mm+cup 81 radius mm (cup 81 edge upward limit line). Diagram (2) shows a line parallel to the X-axis passing through y=y2+10mm (cup 81 center upward limit line). Diagram (3) shows a line parallel to the X-axis passing through the center of the healthy femoral head (x2,y2). Diagram (4) shows a pelvic reference line (X-axis) connecting the lower ends of the teardrop sigma. Patient K, who is to undergo treatment related to artificial hip joint 8, is generally expected to receive treatment related to artificial hip joint 8 (implant) on either the left or right leg. In this case, the side on which the treatment related to artificial hip joint 8 is performed is referred to as the affected side, and the healthy side on which the treatment related to artificial hip joint 8 is not performed is referred to as the healthy side.

[0059] The processing unit 2 of the information processing device 1 recognizes the shape of the healthy femoral head included in the acquired fluoroscopic image (X-ray image) using, for example, an object detection model such as YOLO or edge detection, and derives the center of the healthy femoral head by calculating the center of curvature from, for example, multiple points located on the outer edge, based on the outer edge which forms an arc shape. The processing unit 2 of the information processing device 1 derives an upward limit line (line segment (2)) of the cup 81 center (center of the femoral head of the implant on the affected side) that is within a predetermined value (for example, 10 mm) in the positive direction (upward direction) on the Y axis with respect to a line (line segment (3)) that passes through the derived healthy femoral head center (x2, y2) and is parallel to the X axis (line segment (4)). The upward limit line of the center of cup 81 (center of the femoral head of the implant on the affected side) serves as a reference line in determining the upward limit line of the periphery of cup 81 (line segment (1)), that is, the upward limit line (line segment (1)) in the reachable area (strike zone).

[0060] The center of the bone on the affected side where the treatment for the artificial hip joint 8 is performed corresponds to the center (x, y) of the cup 81 to be placed. The Y-axis indicates the upward direction in the human body, and the positive direction on the Y-axis indicates upward. The Y coordinate (y) of the center (x, y) of the cup 81 to be placed is set to be in a range not exceeding, for example, 10 mm upward (y < y2 + 10 [mm]) with respect to the Y coordinate (y2) of the center of the bone on the healthy side. Therefore, the processing unit 2 of the information processing device 1 passes through a point that has been moved upward (along the positive direction of the Y-axis) by the upward limit value (for example, 10 mm) with respect to the Y coordinate (y2) of the center of the bone on the healthy side, and derives an upward limit line (line segment (2)) of the cup 81 center (the center of the bone of the implant on the affected side) that is perpendicular to the Y coordinate. Further, a cup 81 edge upward limit line (line segment (1)), that is, an upward limit line (line segment (1)) in the reach area (strike zone), which is parallel to the upward limit line (line segment (2)) of the cup 81 center (the center of the bone of the implant on the affected side) (that is, parallel to the X-axis) and is located upward (in the positive direction on the Y-axis) by the amount of the cup 81 radius, is determined (derived). By deriving the upward limit line (line segment (1): cup 81 edge upward limit line) in the reach area (strike zone) based on the center of the bone on the healthy side in this way, a guideline (the upper limit line in the excavation of the pelvis) when the center of the bone on the affected side (the center of the femoral head of the artificial hip joint 8) is placed can be provided to doctors and the like.

[0061] Based on the derived inner plate line, must-reach line, and upward limit line, the processing unit 2 of the information processing device 1 derives the reach area (strike zone) (S106). Alternatively, the processing unit 2 of the information processing device 1 may derive the reach area (strike zone) based on the derived inner plate line, the position information of the acetabular cup (front and rear wall area information, center area information), and the bone coverage rate (BCI, three-dimensional bone coverage rate). The processing unit 2 of the information processing device 1 derives the area surrounded by these lines as the reach area (strike zone) based on the derived upward limit line, inner plate line, and must-reach line.

[0062] Figure 12 is an explanatory diagram regarding the reachable area (strike zone). This reachable area indicates the region in which the stable fixation of the cup 81 is ensured if a part of the cup 81 is in contact with the area when the cup 81 is placed. Within the reachable area, the upward limit line and the inner plate line indicate the limit lines that must not be exceeded when drilling with the bone drilling reamer 7. By superimposing the thus derived reachable area onto the fluoroscopic image, the positional relationship between the current position of the bone drilling reamer 7 and the region that the bone drilling reamer 7 should reach by drilling (the reachable area) can be provided to the physician in real time by the bone drilling reamer 7 included in the real-time displayed fluoroscopic image.

[0063] The processing unit 2 of the information processing device 1 derives various measured values, target values, and the difference between them at the present time (S107). The processing unit 2 of the information processing device 1 derives the measured values ​​of the cup 81 that are expected to be positioned after the pelvis has been shaved by the bone excavation reamer 7, using fluoroscopic images. The target CE angle, target bone coverage rate, and target distance between the outer edge of the cup 81 and the inner plate line are set in advance as target values ​​and are stored in the storage unit 3 of the information processing device 1 as various values ​​determined in the preoperative plan when performing the procedure on the artificial hip joint 8. The processing unit 2 of the information processing device 1 obtains these various values ​​determined in the preoperative plan by referring to the storage unit 3.

[0064] The processing unit 2 of the information processing device 1 uses the acquired fluoroscopic image to determine the position and inclination of the cup 81 when the pelvis has been shaved by the bone-shaping reamer 7, based on the current position and inclination of the bone-shaping reamer 7 (the outer edge of the hemispherical tip that performs the drilling) in the pelvis of patient K. This inclination includes, for example, the anteversion angle and the abduction angle of the cup 81.

[0065] Figure 13 is an explanatory diagram regarding the anterior tilt angle and abduction angle of the cup 81 (assumed placement when the pelvis has been shaved by the bone-excavating reamer 7). The processing unit 2 of the information processing device 1 identifies three points (A, A', B) of an ellipse using shape recognition processing. In this case, if the angle between A-A' and A'-B in the fluoroscopic image is X, the anterior tilt angle is expressed as "sin^(-1)*tanX" or the like. The processing unit 2 may also recognize the shape and current position of the bone-excavating reamer 7 (the outer edge of the hemispherical tip that performs excavation) corresponding to the assumed placement of the cup 81 in the fluoroscopic image by calculation using the image coordinate system in the fluoroscopic image, and calculate the anterior tilt angle and abduction angle.

[0066] The processing unit 2 of the information processing device 1 may derive the target tilt angle and target abduction angle based on a preset target CE angle. Alternatively, the processing unit 2 of the information processing device 1 may derive the target tilt angle and target abduction angle by referring to a table (target CE angle table) in which the values ​​of the target tilt angle and target abduction angle are associated with each value of the target CE angle. Various lookup tables that the processing unit 2 of the information processing device 1 refers to when performing various calculations, such as the target CE angle table, are stored in the storage unit 3.

[0067] The processing unit 2 of the information processing device 1 outputs various derived support information (inner plate lines, reachable area, difference between measured values ​​and target values) superimposed on and accompanying the perspective image (S108). The support information includes, for example, inner plate lines, reachable area, and the difference between measured values ​​and target values. The processing unit 2 of the information processing device 1 superimposes the reachable area, including the derived inner plate lines, onto the perspective image and outputs it to the display device 41. The processing unit 2 of the information processing device 1 further outputs various measured values, target values, and the difference between these measured values ​​and target values ​​accompanying the perspective image to the display device 41. When outputting this information, the processing unit 2 of the information processing device 1 may generate and output screen data that constitutes a display screen (support information superimposed screen). The processing unit 2 of the information processing device 1 compares the derived measured values ​​(cup CE angle, bony coverage, measured distance between the outer edge of cup 81 and the inner plate line, anterior tilt angle, abduction angle) with the corresponding target values ​​(target CE angle, target bony coverage, target distance between the outer edge of cup 81 and the inner plate line, target anterior tilt angle, target abduction angle) and displays them together with the fluoroscopic image (for example, in a sub-screen or separate frame). This allows physicians and others to efficiently grasp the differences between various measured values ​​and target values ​​at the current time.

[0068] Figure 14 is an explanatory diagram illustrating a support information display screen (support information overlay screen). The processing unit 2 of the information processing device 1 generates screen data constituting the support information overlay screen as a result of the various processes described above, and outputs the screen data to the display device 41. The support information overlay screen includes a fluoroscopic image display area, a acetabular position relationship display area, and a support information display area.

[0069] The fluoroscopic image display area shows a fluoroscopic image (X-ray image) of patient K's pelvis, captured in real time during surgery while the pelvis is being cut with a bone-excavating reamer 7. The reach area, including the inner plate line, is superimposed on the fluoroscopic image.

[0070] The acetabular positional relationship display area displays a fluoroscopic image (X-ray image) of patient K's pelvis, captured in real time during surgery when the pelvis is being cut by a bone-excavating reamer 7. The anterior and posterior walls of the acetabulum are superimposed on the fluoroscopic image. Alternatively, the area where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum is superimposed on the fluoroscopic image. Furthermore, the load-bearing surface area may also be superimposed on the fluoroscopic image.

[0071] The support information display area shows, in list format, the anterior tilt angle, abduction angle, cup CE angle, bony coverage rate, and remaining distance from the inner plate line of the cup 81, assuming the pelvis has been shaved by the bone excavation reamer 7. For these anterior tilt angle, abduction angle, cup CE angle, bony coverage rate, and remaining distance from the inner plate line of the cup 81, the measured values ​​calculated based on the current position and inclination of the bone excavation reamer 7 identified by shape recognition processing in the fluoroscopic image, the values ​​determined in the preoperative plan (target values), and the difference between the measured values ​​and the target values ​​are displayed in list format.

[0072] Figure 15 is an explanatory diagram illustrating the display screen for support information (distortion correction screen). The processing unit 2 of the information processing device 1 generates screen data constituting the distortion correction screen as a result of the various processes described above, and outputs the screen data to the display device 41. The distortion correction screen includes a display area before distortion correction and a display area after distortion correction. In this way, the distortion correction screen displays a comparison between the fluoroscopic image before distortion correction (with support information superimposed) and the fluoroscopic image after distortion correction (with support information superimposed), so that it can present information to doctors and others showing the before and after of distortion correction for the target area (strike zone), and can provide useful information for the doctor when performing a procedure.

[0073] The distortion correction screen may be output on a separate screen from the aforementioned support information superimposed screen, or it may be included in the support information superimposed screen. If the distortion correction screen is output on a separate screen from the support information superimposed screen, the display device 41 that displays the distortion correction screen and the display device 41 that displays the support information superimposed screen may be separate display devices 41. In this case, the information processing device 1 will be connected to two display devices 41: one for displaying the distortion correction screen and another for displaying the support information superimposed screen.

[0074] The pre-distortion correction display area displays the image shown in the perspective image display area of ​​the aforementioned support information superimposed screen (a perspective image with the reachable area superimposed). Furthermore, the image displayed in the pre-distortion correction display area (pre-correction image) shows a grid of auxiliary lines consisting of multiple straight lines used when performing distortion correction.

[0075] The distortion-corrected display area shows the distortion-corrected image (a perspective image with the target area superimposed). Furthermore, the image displayed in the distortion-corrected display area (the corrected image) shows the auxiliary lines used during the distortion correction process in a curved form corresponding to the correction. As a result, the target area (strike zone) is also displayed with a changed shape (the area in the pelvis) according to the distortion correction.

[0076] The processing unit 2 of the information processing device 1 may, for example, perform distortion correction by attaching a template containing straight lines to the fluoroscopic image (image before correction). In this case, the processing unit 2 of the information processing device 1 is a device that converts X-rays that have passed through the human body into a digital image, and may use a flat panel with a distortion correction function.

[0077] Alternatively, the processing unit 2 of the information processing device 1 may perform distortion correction processing on the fluoroscopic image (uncorrected image) according to the distortion aberration determined based on the X-ray imaging characteristics of the X-ray machine 62, etc. The X-ray machine 62 that takes fluoroscopic images (X-ray images) has various characteristics depending on the model or type, and the distortion aberration corresponding to the amount of distortion in the captured fluoroscopic image (X-ray image) can also be identified. The storage unit 3 of the information processing device 1 stores parameters such as distortion aberration corresponding to each of the various X-ray machines 62, and the processing unit 2 of the information processing device 1 may perform distortion correction using parameters such as distortion aberration corresponding to the X-ray machine 62, etc. In this case, it is not necessary to use a flat panel, which is a dedicated device that performs distortion correction by hardware processing, and the distortion-corrected fluoroscopic image can be displayed using a relatively inexpensive display (display device 41 that does not have a hardware-processed distortion correction function).

[0078] According to this embodiment, the processing unit 2 of the information processing device 1 acquires a fluoroscopic image (X-ray image) of patient K who is undergoing treatment related to the artificial hip joint 8. The fluoroscopic image is, for example, an X-ray image. As part of the treatment related to the artificial hip joint 8, when a bone reamer 7 is used to form a bone cavity (installation space) in the pelvis that will serve as a receptacle for installing the cup 81 of the artificial hip joint 8 (excavating the pelvis), the fluoroscopic image (X-ray image) is captured in real time as a video, and the processing unit 2 of the information processing device 1 sequentially acquires the fluoroscopic image (X-ray image) that has been captured in real time. The processing unit 2 of the information processing device 1 recognizes (derives) an inner plate line indicating the maximum cutting point (limit depth) when cutting the pelvis to place the cup 81 containing the artificial hip joint 8, and a required line (line to be exceeded) indicating the minimum cutting point when cutting the pelvis to place the cup 81, in the acquired fluoroscopic image of patient K's pelvis. Alternatively, the processing unit 2 of the information processing device 1 recognizes (derives) acetabular positional relationship information (anterior-posterior wall region information, central region information) that indicates the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81 in the pelvis of patient K, so that the cup 81 can obtain sufficient initial fixation. Information regarding the inner plate line is derived based on the positional information of the inner plate output from the learning model 101 by inputting the acquired fluoroscopic image (X-ray image) into the learning model 101. That is, the positional information of the inner plate from the learning model 101 may correspond to the inner plate line itself, or the positional information of the inner plate from the learning model 101 may be a plurality of points (point cloud) that constitute the inner plate line, and the processing unit 2 of the information processing device 1 may generate the inner plate line using the point cloud. Furthermore, the processing unit 2 of the information processing device 1 derives the cup CE angle or bone coverage ratio (BCI, 3D bone coverage ratio) when the placement of the cup 81 is assumed to be in a state where the pelvis has been shaved by the bone excavation reamer 7 (in real time). The processing unit 2 of the information processing device 1 superimposes the recognized (derived) inner plate lines and required lines onto the fluoroscopic image, and also displays the derived cup CE angle attached to the fluoroscopic image, for example, on a sub-screen, thereby providing useful support information to a physician performing a procedure on patient K regarding the artificial hip joint 8.The processing unit 2 of the information processing device 1 recognizes the shape of the outer edge (the outer edge of the hemispherical tip that performs drilling) of the bone drilling reamer 7 included in the fluoroscopic image (X-ray image) using, for example, an object detection model such as YOLO or edge detection. The processing unit 2 of the information processing device 1 considers the outer edge of the bone drilling reamer 7 as the outer edge of the cup 81 and derives the cup CE angle when the cup 81 is assumed to be positioned in the state where the pelvis has been drilled by the bone drilling reamer 7. The processing unit 2 of the information processing device 1 derives the cup CE angle as the angle formed by the line connecting the intersection of the outer edge of the cup 81 and the white line (Acetabular sourcil) on the pelvic load surface (COR: Center of Rotation) of the cup 81 assumed to be positioned in the state where the pelvis has been drilled by the bone drilling reamer 7, and the line connecting the lower ends of the left and right teardrop marks on the pelvis, which is perpendicular to the pelvic reference line (Y axis). The processing unit 2 of the information processing device 1 recognizes the shape of the lower ends of the left and right teardrops contained in the fluoroscopic image (X-ray image) using, for example, an object detection model such as YOLO or edge detection, and identifies a pelvic reference line passing through these lower ends of the left and right teardrops. The processing unit 2 of the information processing device 1 identifies a vertical line that is perpendicular to the identified pelvic reference line and passes through the center (COR) of the cup 81, which is assumed to be positioned in a state where the pelvis has been shaved by the bone excavation reamer 7. The processing unit 2 of the information processing device 1 derives the cup CE angle by calculating the angle between the vertical line passing through the center (COR) of the cup 81 and the line connecting the intersection point of the outer edge of the cup 81 and the white line (Acetabular sourcil) of the pelvic load surface. The processing unit 2 of the information processing device 1 acquires the fluoroscopic image captured in real time and derives the cup CE angle based on the outer edge of the bone excavation reamer 7 contained in the acquired fluoroscopic image. This allows the physician to be provided with the cup CE angle if the cup 81 were placed at the current time, in accordance with the current position of the bone excavation reamer 7, i.e., the excavation state by the bone excavation reamer 7. In other words, an intraoperative support system S can be configured that enables real-time evaluation of cup placement and coverage during surgery.

[0079] According to this embodiment, the storage unit 3 of the information processing device 1 stores an actual file of a learning model 101 (inner plate line model) that has been trained to output inner plate position information (landmarks) when a fluoroscopic image (X-ray image) is input. The processing unit 2 of the information processing device 1 inputs the acquired fluoroscopic image (X-ray image) to the learning model 101 (inner plate line model), and the learning model 101 (inner plate line model) outputs the inner plate position information superimposed on the fluoroscopic image. Each of the inner plate position information is output as a point (landmark), and by outputting multiple points (landmarks) on the fluoroscopic image, the inner plate line may be identified by forming (connecting) lines between these multiple points (landmarks). By using the learning model 101 (inner plate line model) that has been trained to output inner plate position information (landmarks) when a fluoroscopic image is input in this way, the inner plate line indicating the maximum cutting point when cutting the pelvis to position the cup 81 containing the artificial hip joint 8 can be efficiently identified (derived). The processing unit 2 of the information processing device 1 displays (outputs) the inner plate line, derived using the learning model 101 based on the fluoroscopic image (X-ray image), superimposed on the fluoroscopic image (X-ray image). The fluoroscopic image (X-ray image) displayed in real time during surgery shows the bone excavation reamer 7 at the current moment, and the inner plate line is superimposed on it. This effectively presents the physician or other operator of the bone excavation reamer 7 with the inner plate line (excavation limit line) that marks the maximum excavation point when cutting the pelvis, i.e., the limit point that must not be exceeded.

[0080] According to this embodiment, the learning model 101 is generated by training it using training data. Therefore, the training data used is data in which positional information of the inner plate is annotated to fluoroscopic images (X-ray images) of a patient or subject different from patient K, who will undergo treatment related to the artificial hip joint 8 in this case. In this case, the fluoroscopic images (X-ray images) used in the training data include pseudo-fluoroscopic images (DRR-derived fluoroscopic images [DRR-F]) generated from DRR images (Digital Reconstructed Radiograph: CT reconstruction simulation images) converted or reconstructed using multiple tomographic images (CT images) taken of the subject or the like. The processing unit 2 of the information processing device 1 can align the DRR images converted from tomographic images (CT images) and the pseudo-fluoroscopic images (DRR-F) generated from the DRR images by applying the same internal body coordinate system, and can perform various calculations on these images (CT images, DRR images, and pseudo-fluoroscopic images). The simulated fluoroscopic image (DRR-F) shows the pelvis from multiple different fluoroscopic angles, such as the front (anterior pelvis) and lateral (lateral pelvis). The simulated fluoroscopic image is annotated with an inner platen line (or a point cloud constituting the inner platen line) based on the positional information of the inner platen defined in multiple tomographic images (CT images). In other words, simulated fluoroscopic images (DRR-F) from many directions consisting of multiple different fluoroscopic angles are created, and the inner platen line is annotated not only to the front (anterior pelvis) but also to the simulated fluoroscopic images (DRR-F) that include angles other than the front of the pelvis. Therefore, even if the intraoperative fluoroscopic image is not the front of the pelvis, the inner platen line can be accurately displayed. The simulated fluoroscopic image (DRR-F) corresponds to the problem data in the training data, and the inner platen line annotated on the simulated fluoroscopic image (DRR-F) corresponds to the answer data in the training data. The response data annotated to the pseudo-perspective image (DRR-F) is not limited to inner plate lines, but may also be an outline of the outer edge of the cup 81 at the installation position when the cup 81 is installed, derived based on the inner plate line, the required line, and the cup CE angle.In other words, the training data may be a pseudo-fluoroscopic image (DRR-F) to which the outer edge of the cup 81 at the placement position of the cup 81 is annotated. By generating a pseudo-fluoroscopic image using a DRR image converted from a tomographic image (CT image) in this way, it is possible to obtain a pseudo-fluoroscopic image that is less blurry and relatively clear compared to, for example, an X-ray image directly created from a CT image. When generating a pseudo-fluoroscopic image (DRR-F) from a DRR image, the tomographic image (CT image) may be adjusted or corrected so that the pseudo-fluoroscopic image (DRR-F) is a complete frontal view of the pelvis, and the DRR image may be reconstructed. By generating the pseudo-fluoroscopic image (DRR-F) so that the pelvis included in it is a complete frontal view, angle correction of the X-ray image relative to the pelvis (frontal view of the pelvis) can be performed, and the fluoroscopic image (X-ray image) of patient K acquired during surgery and the pseudo-fluoroscopic image (DRR-F) can be accurately aligned or superimposed. It is possible that multiple pseudo-fluoroscopic images (DRR-F) are generated from the DRR image converted from the tomographic image (CT image) by varying the fluoroscopy angle, i.e., the X-ray irradiation angle on the subject. In other words, since multiple pseudo-fluoroscopic images (DRR-F) with different fluoroscopy angles can be generated from the same DRR image, even if the number of subjects for the tomographic image (CT image) is relatively small, a large number of pseudo-fluoroscopic images (DRR-F) can be generated, thereby increasing the training data. Thus, in the generation stage (learning process) of the learning model 101, tomographic images (CT images) taken of subjects are required, but when the learning model 101 is put into operation, i.e., when the treatment related to the artificial hip joint 8 is performed on patient K using the learning model 101, it is not necessary to take a CT image of patient K (taking a preoperative CT image). Depending on the size of the medical institution performing the artificial hip joint 8 procedure, it is conceivable that they may own an X-ray imaging device for taking fluoroscopic images (X-ray images) but not a CT scanner 61. In such cases, the availability of the intraoperative support system S can be improved by using a learning model 101 that uses only fluoroscopic images (X-ray images) as input data.In other words, by using the intraoperative support system S, when performing a procedure on patient K regarding the artificial hip joint 8, it is possible to display the cup placement angle of the artificial hip joint 8, display the cup placement position information, and correct the angle of the X-ray image relative to the pelvis (anterior view of the pelvis) using only intraoperative fluoroscopic images (X-ray images).

[0081] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0082] With respect to the multiple claims described in the claims, they can be combined with each other regardless of the form of reference. Multiple dependent claims that depend on multiple claims may be described in the claims. Multiple dependent claims that depend on multiple dependent claims may also be described. Even if multiple dependent claims that depend on multiple dependent claims are not described, this does not limit the description of multiple dependent claims that depend on multiple dependent claims. [Explanation of Symbols]

[0083] S Intraoperative Support System K patient 1. Information Processing Device 2 Processing Units 3 Storage section M recording medium P Program (Program Product) 4 Input / Output Interfaces 41 Display device 5 Communications Department 101 Learning Model (Location Information Model) 102 Second Learning Model (Bone Coverage Model) 61. CT scanner (tomographic imaging device) 62. X-ray machine (fluoroscopic imaging device) 7. Reamer for bone excavation 8. Artificial hip joint 81 cups

Claims

1. to the computer Obtain fluoroscopic images of patients undergoing artificial hip joint replacement procedures. A learning model, trained to output positional information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum when a fluoroscopic image is input, is used to derive positional information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic image. Based on the derived positional information of the inner plate, the inner plate line indicating the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint is recognized in the acquired fluoroscopic image of the patient's pelvis. Based on the derived positional information of the anterior and posterior walls of the acetabulum, the acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image is recognized. Using a bone-excavating reamer used in the procedure for the artificial hip joint, the cup CE angle or bony coverage rate is derived when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved. The recognized inner plate line and the acetabular positional relationship information are superimposed on the fluoroscopic image. The derived cup CE angle or bony coverage rate is displayed in conjunction with the fluoroscopic image. A program that executes a process.

2. The acetabular positional relationship information includes anterior and posterior wall region information indicating the areas of the anterior and posterior walls of the acetabular where the front and rear of the cup make contact when the cup is placed. The program according to claim 1.

3. The acetabular positional relationship information includes central region information indicating the area where the center of the cup is located when the cup is positioned in contact with the anterior and posterior walls of the acetabular lid. The program according to claim 1.

4. The aforementioned bone coverage rate includes BCI (Bone Coverage Index) or three-dimensional bone coverage rate. The program according to claim 1.

5. The training data for the aforementioned learning model is generated based on pseudo-fluoroscopic images produced from tomographic images of subjects other than the aforementioned patient. The program according to claim 1.

6. The aforementioned pseudo-fluoroscopic image includes multiple pseudo-fluoroscopic images generated at multiple different fluoroscopic angles relative to the DRR image converted from the tomographic image. The program according to claim 5.

7. Based on the center of the femoral head on the healthy side that does not undergo the artificial hip joint procedure, the upward limit line is derived relative to the center of the femoral head on the affected side that undergoes the artificial hip joint procedure. Based on the derived upward limit line, the inner plate line, and the acetabular positional relationship information, the reachable area is derived. The derived destination region is superimposed onto the perspective image. The program according to claim 1.

8. Based on a predetermined target CE angle or target bony coverage ratio to stabilize the cup that is to be placed, the intersection point between the outer edge of the cup and the inner edge of the load-bearing surface on the patient's pelvis, which serves as a reference when ensuring the target CE angle or target bony coverage ratio, and the radius of the cup, a minimum cutting line is derived in the patient's pelvis to indicate the minimum cutting point when cutting the pelvis to place the cup. The program according to claim 1.

9. The derived cup CE angle or bone coverage rate is compared with the target CE angle or target bone coverage rate set in advance as a target value and output. When the patient's pelvis is shaved down by the bone-excavating reamer, the actual distance between the outer edge of the cup and the inner plate line is determined by considering the placement of the cup. The target distance between the outer edge of the cup and the inner plate line, which is set in advance as the target value, is derived. The calculated measured distance is compared with the target distance and output. The program according to claim 1.

10. When the position of the cup is assumed to be in the state in which the patient's pelvis has been shaved by the bone-excavating reamer, the anterior tilt angle and abduction angle of the cup are derived. Output the derived forward tilt angle and abduction angle. The program according to claim 1.

11. Depending on the target CE angle or target bony coverage, the target anterior tilt angle and target abduction angle are derived. The derived target tilt angle and target abduction angle are combined and output. The program according to claim 9.

12. The derived forward tilt angle and abduction angle of the cup are compared with the target forward tilt angle and target abduction angle and output. The program according to claim 11.

13. Based on the acquired fluoroscopic image, the current position of the bone drilling reamer in the patient's pelvis is obtained. From among the multiple tomographic images of the patient's pelvis, a tomographic image including the current position of the bone drilling reamer is extracted. The outer edge diagram of the cup, assuming the cup's placement after the patient's pelvis has been shaved by the bone-excavating reamer, is superimposed onto the extracted tomographic image and output. The program according to claim 5.

14. On the computer, Obtain fluoroscopic images of patients undergoing artificial hip joint replacement procedures. A learning model, trained to output positional information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum when a fluoroscopic image is input, is used to derive positional information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic image. Based on the derived positional information of the inner plate, the inner plate line indicating the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint is recognized in the acquired fluoroscopic image of the patient's pelvis. Based on the derived positional information of the anterior and posterior walls of the acetabulum, the acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image is recognized. Using a bone-excavating reamer used in the procedure for the artificial hip joint, the cup CE angle or bony coverage rate is derived when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved. The recognized inner plate line and the acetabular positional relationship information are superimposed on the fluoroscopic image. The derived cup CE angle or bony coverage rate is displayed in conjunction with the fluoroscopic image. An information processing method that executes a process.

15. An acquisition unit that acquires fluoroscopic images of patients undergoing artificial hip joint replacement surgery, A position information derivation unit that derives position information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic image into a learning model that has been trained to output position information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum when a fluoroscopic image is input. Based on the derived positional information of the inner plate, an inner plate line recognition unit recognizes an inner plate line in the patient's pelvis in the acquired fluoroscopic image that indicates the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint, Based on the derived positional information of the anterior and posterior walls of the acetabulum, an acetabular positional relationship information recognition unit recognizes acetabular positional relationship information relating to the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image, A deriving unit for deriving the cup CE angle or bony coverage rate when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved using a bone excavation reamer used in the procedure for the artificial hip joint, A display unit that superimposes the recognized inner plate line and acetabular positional relationship information onto the fluoroscopic image and displays the derived cup CE angle or bony coverage rate attached to the fluoroscopic image. An information processing device equipped with the following features.

16. Acquire fluoroscopic images of patients undergoing artificial hip joint replacement surgery, and training data including positional information of the patient's inner plate, Based on the acquired training data, a learning model is generated that outputs the position information of the inner plate when a perspective image is input. The fluoroscopic image includes multiple pseudo-fluoroscopic images generated at multiple different fluoroscopic angles relative to the DRR image converted from the patient's tomographic image. Methods for generating learning models.

Images