Medical image processing device and method of operating the same

Abstract

To provide a medical image processing device that allows a user to confirm anatomical structures such as blood vessels and nerves within an organ during surgery without burdens by performing display control of a 3D organ model, and a method of operating the same.SOLUTION: A 3D organ model acquisition unit 20 acquires a 3D organ model 25. A hard mirror image acquisition unit 21 acquires a hard mirror image 27 including an organ to be observed. A posture matrix estimation unit 22 estimates a posture matrix representing a posture of the organ to be observed included in the hard mirror image 27, in a three-dimensional coordinate system used for the 3D organ model 25. A display control unit 23 controls the display of the 3D organ model 25 on the basis of the posture matrix.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a medical image processing device that controls the display of an organ to be observed and a 3D organ model corresponding to the organ to be observed, and a method for operating the same. [Background technology]

[0002] In the current medical field, surgery is performed inside the abdominal cavity without laparotomy using rigid endoscope images. To achieve high-quality surgery, it is important to proceed with the surgery according to the preoperative plan while confirming the course of major blood vessels hidden by fat, the course of blood vessels discovered during dissection or transection, their relationship with nearby nerves and organs, and the location of lesions.

[0003] As a document related to the above, Patent Document 1 describes displaying a planar organ image of an organ and a 3D simulated image that is a 3D model that three-dimensionally imitates the organ. Non-Patent Document 1 also describes acquiring 3D liver surface data from intraoperative video and aligning it with a preoperative CT model, and then determining specific landmarks and using the information from those landmarks to track the object being observed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-131741 [Non-patent literature]

[0005] [Non-Patent Document 1] Augmented Reality Navigation for Stereoscopic Laparoscopic Anatomical Hepatectomy of Primary Liver Cancer: Preliminary Experience(http:pubmed.ncbi.nlm.nih.gov / 33842378) Summary of the Invention [Problem to be solved by the invention]

[0006] 3D organ models obtained before surgery are used to confirm the course of blood vessels and their position within organs during surgery. While it is possible to print out the 3D organ model on paper for reference during surgery, or to display the 3D organ model on a display, the former requires the surgeon to mentally compensate for the difference between the printed 3D organ model and how the organ actually appears during surgery, while the latter requires manual manipulation of the display to adjust the 3D organ model to match the actual appearance of the organ during surgery. Both methods place a burden on users, including surgeons and assistants.

[0007] An object of the present invention is to provide a medical image processing apparatus and an operating method thereof that allow a user to check anatomical structures such as blood vessels and nerves during surgery without any burden. [Means for solving the problem]

[0008] The medical image processing device of the present invention has a processor, which acquires a 3D organ model corresponding to the organ to be observed, acquires a medical image including the organ to be observed, estimates an orientation matrix that represents the orientation of the organ to be observed included in the medical image in a three-dimensional coordinate system used for the 3D organ model, and controls the display of the 3D organ model based on the orientation matrix.

[0009] The orientation matrix may be estimated using a medical image and a training model trained using a ground truth orientation matrix for the medical image as input. The processor may also control the rotation of the 3D organ model in a three-dimensional coordinate system.

[0010] The processor may periodically change the orientation of the 3D organ model in the three-dimensional coordinate system when the degree of overlap of structures within the observed organ is equal to or greater than a threshold, or when the orientation of the observed organ has not changed for a certain period of time. The processor may determine the degree of overlap of structures in the 3D organ model and change the orientation of the 3D organ model so that the degree of overlap of structures is within a specific range.

[0011] The processor may calculate a correction amount (e.g., a correction matrix, yaw, pitch, and roll) for correcting the shape difference between a predetermined reference organ and the observed organ, correct the orientation matrix according to the correction amount, and control the display of the 3D organ model based on the corrected orientation matrix. The processor may also estimate the orientation matrix using the 3D organ model in addition to the medical image.

[0012] The processor may also apply a process to the posture matrix that smooths the angular change of the 3D organ model in the time direction. The posture matrix may also be estimated using multiple medical images acquired at different times. The posture matrix may also be estimated using a medical image and a posture matrix estimated before the estimation time.

[0013] The display of the 3D organ model may be controlled in conjunction with the field of view based on the medical image. The processor may recognize the surgical procedure from the medical image and control the display angle of the 3D organ model on or off or the display update rate according to the surgical procedure.

[0014] The processor may determine whether the orientation matrix estimation works from the medical image, and if the orientation matrix estimation does not work, may stop displaying the 3D organ model. The processor may also switch the display mode of the 3D organ model in accordance with a display switching operation.

[0015] The processor may recognize the progress of resection of the observed organ from the observed organ and switch the display mode of the 3D organ model according to the progress of the resection. When the processor recognizes a specific movement, it may activate display control of the 3D organ model using an orientation matrix. The processor may accept user corrections regarding the display orientation of the 3D organ model, and the correction information regarding the corrections may be used for training the learning model or as a correction amount for correcting the shape difference between a predetermined reference organ and the observed organ.

[0016] The medical image may be a rigid endoscope image or an ultrasound image. The 3D organ model may be extracted from a radiological image or an MRI image. The observed organ may be a liver, kidney, pancreas, uterus, or nerve. The processor may estimate a relative positional relationship between an imaging device for capturing the medical image and the observed organ as a display matrix representing the relative positional relationship in a three-dimensional coordinate system, and control the display of the 3D organ model based on the display matrix.

[0017] The operating method of the medical image processing device of the present invention includes the steps of: a processor acquiring a 3D organ model corresponding to the organ to be observed; acquiring a medical image including the organ to be observed; estimating an orientation matrix that represents the orientation of the organ to be observed included in the medical image in a three-dimensional coordinate system used for the 3D organ model; and controlling the display of the 3D organ model based on the orientation matrix. [Effects of the Invention]

[0018] According to the present invention, by controlling the display of a 3D organ model, the user can easily check anatomical structures such as blood vessels and nerves during surgery. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic diagram of a medical image processing system. [Figure 2]FIG. 1 is an image diagram showing a rigid endoscope image and a 3D organ model. [Figure 3] FIG. 2 is an explanatory diagram showing the input and output relationship of a learning model. [Figure 4] FIG. 1 is an explanatory diagram showing a three-dimensional coordinate system consisting of an X-axis, a Y-axis, and a Z-axis. [Figure 5] 1A and 1B are explanatory diagrams showing a 3D organ model before and after display control based on an attitude matrix. [Figure 6] FIG. 10 is an explanatory diagram showing cyclically changing the posture of a 3D organ model. [Figure 7] FIG. 10 is an explanatory diagram showing a process of smoothing the angle change of a 3D organ model over time. [Figure 8] FIG. 1 is an explanatory diagram showing that three rigid endoscope images taken at different times are input into a learning model. [Figure 9] FIG. 10 is an explanatory diagram showing that a rigid endoscope image and a posture matrix before estimation are input to a learning model. [Figure 10] FIG. 1 is an explanatory diagram showing a 3D organ model of the liver with overlapping blood vessels. [Figure 11] 10 is a flowchart showing a method for determining the degree of structural overlap. [Figure 12] FIG. 10 is an explanatory diagram showing a 3D organ model rotated at an optimal angle. [Figure 13] FIG. 10 is an explanatory diagram showing how an attitude matrix is ​​corrected in accordance with a correction matrix. [Figure 14] FIG. 1 is an explanatory diagram showing hepatic segments of a reference liver and hepatic segments of an observation target organ. [Figure 15] FIG. 10 is an explanatory diagram showing the center of gravity, right end, and bottom end of a reference liver and the center of gravity, right end, and bottom end of an observation target organ. [Figure 16] FIG. 1 is an explanatory diagram showing a learning model to which rigid endoscope images and a 3D organ model are input. [Figure 17] FIG. 10 is an explanatory diagram showing the rotation of the angle of a 3D organ model depending on the field of view. [Figure 18] 10 is an explanatory diagram showing the control of the angle display of a 3D organ model and the control of the display update rate according to the surgical procedure. FIG. [Figure 19] 10A and 10B are image diagrams showing rigid endoscope images for which the orientation matrix can be estimated and rigid endoscope images for which the orientation matrix cannot be estimated. [Figure 20] FIG. 10 is an explanatory diagram showing how the angle of a 3D organ model is changed by a display switching operation. [Figure 21] FIG. 10 is an explanatory diagram showing how the angle of a 3D organ model is changed according to the progress of resection of the organ. [Figure 22] FIG. 10 is an image diagram showing a 3D organ model when a specific action is recognized and a 3D organ model when no specific action is recognized. [Figure 23] FIG. 10 is an explanatory diagram showing how learning is performed based on angle correction of a 3D organ model by a user. [Figure 24] 10 is a flowchart showing a series of steps for controlling the display of a 3D organ model based on an orientation matrix. DETAILED DESCRIPTION OF THE INVENTION

[0020] 1, the medical image processing system 10 includes a rigid scope 11 and a medical image processing device 12. The rigid scope 11 captures images of the inside of the body of a patient P and transmits the rigid scope images obtained by the capture to the medical image processing device 12. The rigid scope 11 is also connected to a light source device (not shown), and illumination light from the light source device is supplied to the rigid scope 11.

[0021] The medical image processing device 12 is a computer such as a server, and is connected to a display 15 and a user interface 16. The medical image processing device 12 is also connected to a network NT. A PACS (Picture Archiving and Communication System) and the like are connected to the network NT, and various image data and the like from the PACS are imported into the medical image processing device 12 via the network NT.

[0022] In the medical image processing device 12, programs for executing various processes are stored in a program memory (not shown). A central control unit (not shown) consisting of a processor executes the programs in the program memory, causing the medical image processing device 12 to realize the functions of a 3D organ model acquisition unit 20, a rigid endoscope image acquisition unit 21, a posture matrix estimation unit 22, and a display control unit 23.

[0023] The 3D organ model acquisition unit 20 acquires a 3D organ model corresponding to the organ to be observed. The 3D organ model is acquired from a 3D organ model image server (not shown) or the like via the network NT. The 3D organ model is a model extracted from a radiation image such as an X-ray image or a CT image, or from an MRI. As shown in FIG. 2, the 3D organ model is configured as a 3D organ model 25 in which structures such as blood vessels are superimposed on the organ to be observed. Specifically, when the organ to be observed is the liver, a model in which the internal blood vessels 25b of the liver are superimposed on the liver 25a is displayed on the display 15. Note that the organ to be observed may be, for example, a kidney, pancreas, spleen, uterus, or nerve, in addition to the liver, and may be any other organ, without being limited to the above.

[0024] The rigid endoscope image acquisition unit 21 acquires a rigid endoscope image including an organ to be observed. The rigid endoscope image is acquired from the rigid endoscope 11. Specifically, when the organ to be observed is the liver, as shown in FIG. 2, the rigid endoscope image 27 includes not only the liver 27a but also surrounding structures 27b and an ultrasound probe 27c, which is one of various treatment tools. The rigid endoscope image 27 is displayed in parallel with the 3D organ model 25. Note that in this embodiment, a rigid endoscope image is used as the medical image, but the medical image is not limited to this. For example, an ultrasound image can also be used as the medical image.

[0025] The orientation matrix estimation unit 22 estimates an orientation matrix that represents the orientation of the observation target organ included in the rigid endoscope image in a three-dimensional coordinate system used for the 3D organ model. The orientation matrix estimation will be described in detail below. The display control unit 23 controls the display of the 3D organ model based on the orientation matrix. Specifically, as shown in FIG. 2, the display control unit 23 controls the display of the liver and the blood vessels inside the liver according to the orientation matrix. This updates the display of the 3D organ model in conjunction with the orientation of the observation target organ, thereby reducing the burden on doctors and assistants and facilitating their understanding of the anatomy of the surgical field. This contributes to the realization of high-quality surgery. The display control based on the orientation matrix will also be described in detail below. It is also possible to estimate the relative positional relationship between the observation target organ and an imaging device for capturing medical images, such as an ultrasound probe, in addition to the rigid endoscope 11, as a display matrix that represents the relative position in a three-dimensional coordinate system, and control the display of the 3D organ model based on the display matrix. The display matrix estimation is performed by a display matrix estimation unit (not shown) executed by a processor. The relative positional relationship between the imaging device and the organ to be observed is determined, for example, by the movement of the organ to be observed, the enlargement or reduction of the medical image, or translation.

[0026] In this embodiment, the angle of the organ is estimated from the rigid endoscope image, rather than acquiring the imaging position and estimating the angle of the organ, as in Patent Document 1. Patent Document 1 assumes a static imaging target, and therefore regards the imaging position as the orientation of the organ. Therefore, unlike this embodiment, dynamic imaging targets in dynamically changing rigid endoscope images are not considered. Furthermore, when 3D data is superimposed on video during surgery, as in Non-Patent Document 1, it is difficult to grasp the sense of depth, and the superimposition may obscure important structures. However, in this embodiment, the 3D organ model 25 and the rigid endoscope image 27 are displayed side by side, so the structure is not obscured. Furthermore, because the shape of the organ is deformed and dissected into a non-rigid body, when 3D data is superimposed on video during surgery, as in Non-Patent Document 1, the shapes of the two models do not match, making it difficult to understand the correspondence. On the other hand, in the case of this embodiment, the 3D organ model 25 and the rigid endoscope image 27 are displayed side by side from the beginning, and the display of the 3D organ model is updated in conjunction with the posture of the organ being observed, so there is little chance that the relationship between the organ being observed and the 3D organ model will become difficult to understand.

[0027] The estimation of the posture matrix will be explained below. As shown in FIG. 3, the estimation of the posture matrix requires the following steps: A learning model 30 is used, which is trained using rigid endoscope images and a correct posture matrix for the rigid endoscope images as input. Rigid endoscope images actually obtained from the rigid endoscope 11 are input to the learning model, and a posture matrix 31 is output from the learning model. The three-dimensional coordinate system representing the posture matrix is ​​represented by three axes: X, Y, and Z, as shown in FIG. 4 . The X axis is zero on the right side of the patient P and positive values ​​increase toward the left side. The Y axis is zero on the ventral side of the patient P and positive values ​​increase toward the back side. The Z axis is zero on the head side of the patient P and positive values ​​increase toward the feet side. The three-dimensional coordinate system may be a Cartesian coordinate system such as X, Y, and Z axes, or a polar coordinate system represented by a radius vector and a deflection angle, and is not particularly limited. Instead of the learning model 30, the posture matrix may be estimated using multiple models, such as a model that extracts features from rigid endoscope images and a model that estimates the posture matrix from the features.

[0028] Display control based on the orientation matrix is ​​described below. As shown in FIG. 5, 3D organ model 32 represents the model before display control based on the orientation matrix, and 3D organ model 33 represents the model after display control based on the orientation matrix. Because the orientation matrix is ​​estimated based on the orientation of the observed organ contained in the rigid endoscope image, the display of the 3D organ model also changes in accordance with changes in the orientation of the observed organ. Specifically, display controller 23 controls the rotation of the 3D organ model in a three-dimensional coordinate system. This eliminates the need for manual alignment and compensates for differences in appearance between the organ during surgery and the 3D organ model. The rotation axis AX can be set based on the orientation matrix or arbitrarily. For example, the user's line of sight (the line connecting the center of gravity of the observed organ and the center of the field of view of the rigid endoscope) can be used as the rotation axis.

[0029] In the case of Figure 5, when checking how the blood vessel V2 runs below the large blood vessel V1, the user moves the position of the rigid endoscope 11 to a position where the blood vessel V2 can be seen, and a 3D organ model that matches the appearance of the organ in the rigid endoscope image is displayed, allowing the user to check how the blood vessel V2 runs below the large blood vessel V1.

[0030] The display control unit 23 may periodically change the orientation of the 3D organ model in a three-dimensional coordinate system. In this case, instead of display control based on the orientation matrix, the display control is performed according to input from a cyclic operation ON-OFF module. The input from the cyclic operation ON-OFF module is performed using the user interface 16. Examples of the user interface 16 include user input, gestures on the rigid endoscope 11, and voice recognition. Automatic control may be performed, such as turning the cyclic operation ON a few seconds after the rigid endoscope 11 stops moving and turning it OFF when the rigid endoscope 11 starts moving. The preoperative plan may determine whether a specified structure is visible and automatically activate the cyclic operation if it is not visible. The cyclic change in orientation may be a circular orbit around the line of sight, or may be vertical or horizontal. The orientation of the 3D organ model may be periodically changed if the degree of overlap between structures within the observed organ is equal to or greater than a threshold, or if the orientation of the observed organ has not changed for a certain period of time. The overlap degree may be calculated using the method described below.

[0031] Specifically, as shown in FIG. 6, when a 3D organ model 34a is displayed at a predetermined angle, if the periodic operation ON / OFF module sets the periodic operation ON, after a certain time Tp, a 3D organ model 34b is displayed, which is obtained by rotating the 3D organ model 34a by 10° around the rotation axis. Also, after a certain time Tp, a 3D organ model 34c is displayed, which is obtained by rotating the 3D organ model 34a by 20° around the rotation axis. Then, after a certain time Tp, a 3D organ model 34d is displayed, which is obtained by rotating the 3D organ model 34c by -10° around the rotation axis. Also, after a certain time Tp, the 3D organ model is rotated by -20° around the rotation axis, and the original 3D organ display model 34a is displayed. By repeating the above process, the posture of the 3D organ model changes periodically. The periodic change may be a circular orbit.

[0032] When display control is performed based on the posture matrix, the display control unit 23 may perform processing on the posture matrix to smooth the angular change of the 3D organ model over time in order to suppress abrupt changes in the posture of the 3D organ model. Specifically, in the case of FIG. 7A, display control based on the posture matrix updates the display of the 3D organ model 35a at time Ta to a 3D organ model 35b rotated 20° at time Tc after time Ta. In this case, the posture matrix is ​​subjected to processing to smooth the display over time in order to suppress the 20° angular change of the 3D organ model. In this case, as shown in FIG. 7B, the 3D organ model 36a at time Ta is updated to a 3D organ model 36b rotated 10° at time Tb between times Ta and Tb, and then updated to a 3D organ model 36b rotated 20° from the 3D organ model 36a at time Tc after time Tb. This allows the 20° angular change of the 3D organ model to be gradual between times Ta and Tc.

[0033] To improve the accuracy of posture estimation, the posture matrix may be estimated using multiple rigid endoscope images acquired at different times. Specifically, as shown in Fig. 8, rigid endoscope image 38a acquired at time T1, rigid endoscope image 38b acquired at time T2 different from time T1, and rigid endoscope image 38c acquired at time T3 different from times T1 and T2 are input to learning model 30, which outputs posture matrix 31. This is expected to result in more accurate posture estimation of the movement of the 3D organ model, which is displayed and updated in conjunction with the posture of the observed organ, compared to when estimation is performed using only rigid endoscope images acquired at a single time.

[0034] To improve the accuracy of posture estimation, the posture matrix may be estimated using a rigid endoscope image and a posture matrix estimated before the estimation. Specifically, as shown in FIG. 9, a rigid endoscope image 27 and a posture matrix 40 before estimation are input to a learning model 30. In accordance with this input, a posture matrix 31 is output from the learning model 30. When the display of the 3D organ model is controlled based on the output posture matrix, a more accurate posture estimation of the movement of the 3D organ model can be expected compared to when estimation is performed using only a rigid endoscope image from a single timing.

[0035] In the display control based on the posture matrix, the display control unit 23 may determine the degree of overlap of structures in the 3D organ model and change the posture of the 3D organ model so that the degree of overlap of structures falls within a specific range. As shown in Fig. 10, in the 3D organ model 25, many blood vessels 25b are displayed intersecting and overlapping with each other. When blood vessels overlap, it is difficult to see how the blood vessels run. Therefore, it is also possible to display the 3D organ model with as few blood vessels as possible overlapping with each other.

[0036] Therefore, as shown in Figure 11, once the initial posture of the 3D organ model is determined based on the posture matrix, the degree of overlap of structures such as blood vessels is determined from the 3D organ model in the initial posture. If the degree of overlap is below a threshold, the 3D organ model is displayed based on the initial posture. On the other hand, if the degree of overlap exceeds the threshold, the initial posture is slightly changed, and the degree of overlap of structures is determined again from the changed 3D organ model. This is repeated until the degree of overlap becomes below the threshold, and the 3D organ model is displayed based on the posture where the degree of overlap becomes below the threshold. As described above, by adjusting the posture based on the degree of overlap of structures, it is possible to display a 3D organ model 42 in which the 3D organ model 25 is rotated at an optimal angle that minimizes the degree of overlap of structures, as shown in Figure 12.

[0037] The degree of overlap of structures may be determined by determining the degree of overlap between a specific structure designated by the user before or during surgery and other structures. Furthermore, when determining the degree of overlap of structures, the structures may be grouped, and when determining the degree of overlap of structures, structures within the same group may not be determined to be overlapping. When grouping structures, for example, if the structures are blood vessels, each branch of the blood vessel may be grouped into a separate structure. The designation of a specific structure may be performed using the user interface 16.

[0038] The 3D organ model, whose display is controlled based on the orientation matrix, is extracted from radiographic or MRI images of patient P. Therefore, there are differences in imaging conditions and shape differences between patients. Therefore, as shown in FIG. 13 , the correction matrix generation unit 43 calculates a correction matrix 46 for correcting the shape difference between a predetermined reference organ 44 and an observation target organ 45. The matrix correction unit 47 then corrects the orientation matrix 31 according to the correction matrix 46. The display control unit 23 controls the display of the 3D organ model based on the corrected orientation matrix 48 (corrected orientation matrix). This enables the display of a 3D organ model with the shape difference corrected. The correction matrix generation unit 43 and the matrix correction unit 47 are realized by a central control unit (including a processor) executing a program in a program memory in the medical image processing device 10. In this embodiment, the correction matrix 46 is used as a correction amount for correcting the shape difference between the reference organ 44 and the observation target organ 45. However, the orientation matrix 31 may also be corrected using correction amounts such as yaw, pitch, and roll.

[0039] Specifically, as shown in Figures 14(A) and 14(B), the method of calculating the correction matrix involves performing linear registration between the center of gravity Gx of each liver segment of a reference liver 44, which is one of the reference organs, and the center of gravity Gy of each liver segment of an observation target liver 45, which is the organ to be observed, and calculating the correction matrix from the registration linear parameters. Also, as shown in Figures 15(A) and 15(B), the correction matrix is ​​calculated based on the positional relationship between the center of gravity Gp, right end Rp, and bottom end Up of the reference liver 44 and the center of gravity Gq, right end Rq, and bottom end Uq of the observation target liver 45. Methods for calculating the correction matrix include a method of calculating a correction matrix based on the positional relationship of the centers of gravity, etc., as well as a method using a correction model, which will be described later. However, the method is not limited to these, and other correction methods may also be used.

[0040] 16, instead of the correction matrix generation unit 43 and the matrix correction unit 47, a corrected posture estimation model 50 may be used to calculate a posture matrix 31 that matches the patient. In this case, the corrected posture estimation model 50 estimates the posture matrix 31 using a 3D organ model 25 in addition to the rigid endoscope image 27. By using the 3D organ model of the patient P, who is the subject of the rigid endoscope image, the estimated posture matrix matches the shape, etc., of the patient.

[0041] In the above embodiment, the display controller 23 controls the display of the 3D organ model in conjunction with the posture of the organ being observed, but the display controller 23 may also control the display of the 3D organ model in conjunction with the field of view based on the rigid endoscope image. As shown in Fig. 17, when the rigid endoscope image 27 is in field of view A and the 3D organ model 52a is displayed at a specific angle, if field of view A changes to field of view B, the display changes to 3D organ model 52b, which is rotated by an angle corresponding to the movement from field of view A to field of view B.

[0042] In the above embodiment, the surgical procedure recognition unit recognizes the surgical procedure from the rigid endoscope image, and the display control unit 23 turns on and off the angle display control of the 3D organ model according to the surgical procedure. , or the display update speed may be controlled. As shown in FIG. 18, when the surgical procedure is divided into multiple steps, such as surgical procedure A, surgical procedure B, ..., and surgical procedure X, the angle display control of the 3D organ model is turned ON during surgical procedure A and surgical procedure X, and the display control of the 3D organ model is turned OFF during surgical procedure B. When the angle display control of the 3D organ model is turned ON, display control is performed based on the orientation matrix, and the angle of the 3D organ model is controlled in conjunction with the orientation of the organ being observed. On the other hand, when the angle display control of the 3D organ model is turned OFF, display control based on the orientation matrix is ​​not performed, and therefore the display of the 3D organ model is maintained, but the angle control of the 3D organ model is not performed. The surgical procedure recognition unit is realized by a central control unit consisting of a processor in the medical image processing device 10 executing a program in a program memory.

[0043] Furthermore, when the angle display control of the 3D organ model is turned ON, the display update speed of the 3D organ model is set to S1 in surgical procedure A, while the display update speed of the 3D organ model is set to S2, which is different from S1, in surgical procedure B. Since each surgical procedure is different, changing the display update speed of the 3D organ model accordingly makes it possible to display a 3D organ model suited to each surgical procedure.

[0044] In the above embodiment, an inference feasibility determination unit (not shown) may determine whether or not the posture matrix can be estimated from the rigid endoscope image, and the display control unit 23 may stop displaying the 3D organ model if the posture matrix estimation does not work. As shown in FIG. 19(A), if the inference feasibility determination unit determines that the posture matrix can be estimated from the rigid endoscope image 54a, the display control unit 23 controls the display of the 3D organ model 55a based on the posture matrix. On the other hand, as shown in FIG. 19(B), if the inference feasibility determination unit determines that the posture matrix cannot be estimated from the rigid endoscope image 54b, the display control unit 23 stops displaying the 3D organ model 55b (the dotted line indicates that the display is stopped). The inference feasibility determination unit is realized by a central control unit including a processor executing a program in a program memory in the medical image processing device 10. Images for which posture matrix estimation does not work include enlarged scenes and scenes containing smoke.

[0045] In the above embodiment, the angular rotation of the 3D organ model is controlled based on the posture matrix. However, instead, the display mode of the 3D organ model may be switched in accordance with a display switching operation. The display switching operation is performed using the user interface 16. Specifically, as shown in Fig. 20, when a display switching operation X is performed, a resected organ model (e.g., a resected liver model) created in advance in a preoperative resection simulation is displayed as the 3D organ model 57a.

[0046] In the above embodiment, a progress recognition unit (not shown) may recognize the progress of resection of the observation target organ from the observation target organ, and the display control unit 23 may switch the display mode of the 3D organ model according to the progress of resection. As shown in FIG. 21, when the resection progress is K, L, or M, the displayed 3D organ model is changed to different 3D organ models 58a, 58b, and 58c, respectively. For example, 3D organ model 58a is a resected 3D organ model in which a specific cross section has been resected. The progress recognition unit is realized by a central control unit consisting of a processor executing a program in a program memory in the medical image processing device 10.

[0047] In the above embodiment, the motion recognition unit (not shown) may activate display control of the 3D organ model using the posture matrix when it recognizes a specific motion. The specific motion may be a user motion such as a gesture. In the case of FIG. 22(A), the motion recognition unit recognizes a specific motion and activates display control of the 3D organ model 25 (the activated state is represented by a solid line). Here, activation of display control of the 3D organ model 25 refers to updating the display of the 3D organ model 25. On the other hand, in the case of FIG. 22(B), if the motion recognition unit does not recognize a specific motion, display control of the 3D organ model 25 is stopped (the stopped state is represented by a dotted line). Here, stopping display control of the 3D organ model 25 refers to maintaining the display of the 3D organ model 25 and not updating the display of the 3D organ model 25. In this case, angle rotation based on the posture matrix is ​​not performed. The motion recognition unit is implemented in the medical image processing device 10 by a central control unit including a processor executing a program in a program memory.

[0048] In the above embodiment, the correction receiving unit accepts user corrections regarding the display orientation of the 3D organ model, and the correction information regarding the correction may be used for learning the learning model or as a correction amount for correcting the shape difference between a predetermined reference organ and the observed organ. As shown in FIG. 23 , even when the angle of the 3D organ model is controlled based on the orientation matrix, the angle may be rotated using the user interface 16. In this case, the angle rotation is recognized as angle correction information 60, and a corrected orientation matrix 62 is generated according to the angle correction information 60. The learning model 30 uses this corrected orientation matrix 62 for learning, thereby enabling more accurate angle control of the 3D organ model. The angle correction information 60 may also be used as the correction amount described above. Specifically, the correction amount may be reflected in the orientation matrix and used as the display matrix.

[0049] Next, the display control of the 3D organ model will be explained with reference to the flowchart in Figure 24. The 3D organ model acquisition unit 20 acquires a 3D organ model corresponding to the organ to be observed via a network NT or the like. The rigid endoscope image acquisition unit 21 acquires rigid endoscope images obtained by the rigid endoscope 11 from the rigid endoscope 11. The rigid endoscope images and the 3D organ model are each displayed on the display 15. The orientation matrix estimation unit 22 estimates an orientation matrix that represents the orientation of the organ to be observed contained in the rigid endoscope image in a three-dimensional coordinate system used for the 3D organ model. The display control unit 23 controls the display of the 3D organ model based on the orientation matrix.

[0050] In the above embodiment, the hardware structure of processing units that perform various processes, such as the 3D organ model acquisition unit 20, the rigid endoscope image acquisition unit 21, the posture matrix estimation unit 22, the display control unit 23, the correction matrix generation unit 43, and the matrix correction unit 47, is the following various processors: The various processors include a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), a programmable logic device (PLD), a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration specifically designed for performing various processes.

[0051] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0052] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0053] 10 Medical image processing system 11 Rigid scope 12 Medical image processing equipment 15 Display 16 User Interface 20. 3D Organ Model Acquisition Department 21 Rigid scope image acquisition unit 22 Posture matrix estimator 23 Display control unit 25 3D organ models 25a Liver 25b Blood vessels 27 Rigidoscopy image 27a Liver 27b Structure 27c Ultrasound Probe 30 Learning Model 31 Posture matrix 32, 33 3D organ models 34a, 34b, 34c, 34d 3D organ models 35a, 35b 3D organ models 36a, 36b, 36c 3D organ models 38a, 38b, 38c Rigidoscopy images 40 Pre-estimation posture matrix 42 (optimal angle rotation) 3D organ model 43 Correction matrix generator 44 Reference Organs 45 Organs to be observed 46 Correction Matrix 47 Matrix correction section 48 Corrected posture matrix 50 Corrected posture estimation model 52a 3D organ model 52b (view-linked rotation) 3D organ model 54a Rigidoscope image (possible estimation) 54b Rigid scope image (cannot be estimated) 55a, 55b 3D organ models 57c 3D organ model (excised organ model) 58a, 58b, 58c 3D organ models 60 Angle correction information 62 Modified posture matrix P patient AX Rotation Axis V1, V2 blood vessels Gx, Gy Center of gravity Gp, Gq Center of gravity Rp, Rq right end Up, Uq lower end

Claims

1. a processor; The processor: obtaining a 3D organ model corresponding to the organ to be observed; acquiring a medical image including the organ to be observed; Estimating an orientation matrix that represents the orientation of the observation target organ included in the medical image in a three-dimensional coordinate system used for the 3D organ model; A medical image processing device that controls the display of the 3D organ model based on the orientation matrix.

2. The medical image processing apparatus according to claim 1 , wherein the orientation matrix is ​​estimated using a learning model that is trained using the medical image and a correct orientation matrix for the medical image as input.

3. The medical image processing apparatus according to claim 1 , wherein the processor controls rotation of the 3D organ model in the three-dimensional coordinate system.

4. 2. The medical image processing device according to claim 1, wherein the processor periodically changes the orientation of the 3D organ model in the three-dimensional coordinate system when the degree of overlap of structures within the organ to be observed is equal to or greater than a threshold value, or when the orientation of the organ to be observed has not changed for a certain period of time.

5. The medical image processing apparatus according to claim 1 , wherein the processor determines the degree of overlap of structures in the 3D organ model, and changes the orientation of the 3D organ model so that the degree of overlap of the structures falls within a specific range.

6. the processor calculates a correction amount for correcting a shape difference between a predetermined reference organ and the observation target organ, and corrects the posture matrix in accordance with the correction amount; The medical image processing apparatus according to claim 1 , wherein the display of the 3D organ model is controlled based on the corrected orientation matrix.

7. The medical image processing apparatus according to claim 1 , wherein the processor estimates the orientation matrix using the 3D organ model in addition to the medical image.

8. The medical image processing apparatus according to claim 1 , wherein the processor applies a process to the posture matrix to smooth the angular change of the 3D organ model in the time direction.

9. The medical image processing apparatus according to claim 1 , wherein the orientation matrix is ​​estimated using a plurality of medical images obtained at different times.

10. The medical image processing apparatus according to claim 1 , wherein the orientation matrix is ​​estimated using the medical image and an orientation matrix estimated before the estimation is performed.

11. The medical image processing apparatus according to claim 1 , wherein the display of the 3D organ model is controlled in conjunction with a field of view based on the medical image.

12. The medical image processing device according to claim 1 , wherein the processor recognizes a surgical procedure from the medical image, and controls ON / OFF of display control of the angle of the 3D organ model or a display update rate in accordance with the surgical procedure.

13. The medical image processing device according to claim 1 , wherein the processor determines whether the orientation matrix estimation works from the medical image, and stops displaying the 3D organ model if the orientation matrix estimation does not work.

14. The medical image processing apparatus according to claim 1 , wherein the processor switches the display mode of the 3D organ model in accordance with a display switching operation.

15. The medical image processing apparatus according to claim 1 , wherein the processor recognizes the progress of resection of the observation target organ from the observation target organ, and switches the display mode of the 3D organ model according to the progress of the resection.

16. The medical image processing apparatus according to claim 1 , wherein the processor activates display control of the 3D organ model using the orientation matrix when a specific movement is recognized.

17. 3. The medical image processing device of claim 2, wherein the processor accepts user corrections regarding the display direction of the 3D organ model, and the correction information regarding the corrections is used for training the learning model or as a correction amount for correcting shape differences between a predetermined reference organ and an organ to be observed.

18. 18. The medical image processing apparatus according to claim 1, wherein the medical image is either a rigid endoscope image or an ultrasound image.

19. 18. The medical image processing apparatus according to claim 1, wherein the 3D organ model is extracted from a radiological image or an MRI image.

20. 18. The medical image processing apparatus according to claim 1, wherein the organ to be observed is any one of a liver, a kidney, a pancreas, a uterus, and a nerve.

21. 18. A medical image processing device according to claim 1, wherein the processor estimates the relative positional relationship between an imaging device for capturing medical images and an organ to be observed as a display matrix representing the relative positional relationship in a three-dimensional coordinate system, and controls the display of a 3D organ model based on the display matrix.

22. The processor obtaining a 3D organ model corresponding to the organ to be observed; acquiring a medical image including the organ to be observed; estimating an orientation matrix that represents the orientation of the target organ included in the medical image in a three-dimensional coordinate system used for the 3D organ model; and controlling the display of the 3D organ model based on the orientation matrix.

Images