Information processing device, information processing method, and program
By generating computer graphics data and evaluating the stability and consistency of the recognizer, ensuring that the surgical tool identifier fully learns new surgical tools in the new environment, solving the problem of low recognition accuracy and achieving reliable identification of surgical tools.
Patent Information
- Application Number
- CN202380020744.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-15
- Filing Date
- 2023-01-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing surgical tool identifiers are difficult to properly identify new surgical tools after introducing a new environment, and there is a lack of methods to confirm whether the identifier is fully learned, resulting in low recognition accuracy and affecting the surgical process.
By generating computer graphic data, learning based on images of new surgical tools, using evaluation units to calculate the stability and consistency of recognition results, continuing to learn or terminating learning based on the evaluation values, ensuring that the recognizer updates after fully identifying the new surgical tools .
It effectively improves the accuracy of the identification of new surgical tools by the surgical tool recognizer, ensures that the new surgical tools can be correctly identified during the operation, and reduces the trouble caused by insufficient recognition.
Smart Images

Figure CN119031876A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program for updating a surgical tool identifier to become usable after confirming that the surgical tool identifier is in a state capable of recognizing a new surgical tool. Background Art
[0002] As a medical support technology based on a surgical image, there has been proposed a technology in which a surgical tool as a tool used for surgery is recognized based on the surgical image and support based on the recognition result is performed.
[0003] Here, as a technology for recognizing a surgical tool, a technology related to a surgical tool identifier that recognizes a surgical tool from a surgical image achieved by machine learning such as a deep neural network (DNN) is expected to be a technology responsible for medical support.
[0004] The surgical tool identifier is implemented by learning based on a learning model including a learning data set using pre-used surgical tools.
[0005] However, in general, surgical tools having the same functions cannot be fully matched to each environment (hospital) to be introduced.
[0006] Therefore, there are cases where the surgical tool identifier cannot appropriately identify a new surgical tool due to the influence of the introduction environment, etc., and each time a new surgical tool appears, it is necessary to generate a corresponding new learning model and perform learning using the new learning model.
[0007] Therefore, a technique for creating a learning data set required for recognizing a new object has been proposed (see Patent Document 1).
[0008] Citation List
[0009] Patent Literature
[0010] Patent Document 1: Japanese Patent Application Publication No. 2021-047839 Summary of the invention
[0011] Problems to be solved by the present invention
[0012] However, even if a new learning model corresponding to a new surgical tool is generated by the technology described in Patent Document 1 and a surgical tool identifier is learned using the new learning model, there is no method for confirming whether the new surgical tool can actually be recognized after learning and updating the surgical tool identifier.
[0013] The present disclosure is made in view of such a situation, and in particular, after confirming that the surgical tool identifier becomes in a state capable of fully recognizing a new surgical tool without being affected by the introduction environment, the surgical tool identifier can be updated and used.
[0014] Solution to the problem
[0015] According to one aspect of the present disclosure, an information processing device and a program are an information processing device including: a computer graphics (CG) data generating unit, which is configured to generate CG data based on an image including a new surgical tool, and the CG data is used as learning data for learning an identifier for identifying the new surgical tool; a learning unit, which is configured to enable the identifier for identifying the new surgical tool to learn based on the CG data; and an evaluation unit, which is configured to calculate an evaluation value that evaluates a recognition result of the new surgical tool by the identifier based on the image, wherein the learning unit continues the learning of the identifier based on the evaluation value, and a program.
[0016] An information processing method according to one aspect of the present disclosure is an information processing method comprising the following steps: generating (CG) data based on an image including a new surgical tool, the (CG) data being used as computer graphics for learning data for learning an identifier for recognizing the new surgical tool; learning the identifier for recognizing the new surgical tool based on the CG data; and calculating an evaluation value, the evaluation value evaluating a recognition result of the new surgical tool by the identifier based on the image, wherein the learning of the identifier is continued based on the evaluation value.
[0017] In one aspect of the present disclosure, computer graphics (CG) data is generated based on an image including a new surgical tool, the (computer graphics CG) data is used as learning data for learning a recognizer for recognizing the new surgical tool, the recognizer for recognizing the new surgical tool is learned based on the CG data, an evaluation value is calculated, and the evaluation value evaluates the recognition result of the new surgical tool by the recognizer based on the image. The learning of the recognizer is continued based on the evaluation value. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 is a diagram for describing a method for learning a surgical tool identifier.
[0019] Figure 2 is a diagram for describing the recognition result when the learning of the surgical tool identifier is insufficient.
[0020] Figure 3 is a diagram for describing an example in which a 3DCG model is different from a surgical image of a surgical tool.
[0021] Figure 4 is a diagram for describing an overview of the present disclosure.
[0022] Figure 5 is a diagram for describing a configuration example of a surgical image display system of the present disclosure.
[0023] Figure 6 is a diagram for describing a display example of a feedback image.
[0024] Figure 7 This is a diagram for describing a method of acquiring three-dimensional point group information of a new surgical tool.
[0025] Figure 8 This is a diagram for describing a method of generating a 3DCG model from three-dimensional point group information.
[0026] Fig. 9 is a diagram for describing a method of learning a new surgical tool registration identifier.
[0027] Fig.10 This is a diagram used to describe a method for recreating a 3DCG model.
[0028] Fig.11 is a diagram for describing a method for calculating a stability evaluation value.
[0029] Fig.12 is a diagram for describing a method for calculating a consistency evaluation value.
[0030] Fig.13 is a flowchart for describing a new surgical tool registration process.
[0031] Fig.14 is a flowchart for describing a modification of the new surgical tool registration process.
[0032] Fig.15 A configuration example of a general-purpose computer is shown. DETAILED DESCRIPTION
[0033] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.
[0034] Hereinafter, a mode for carrying out the present technology will be described. The description will be given in the following order
[0035] 1. Overview of the Disclosure
[0036] 2. Advantageous Implementation Methods
[0037] 3. Modification of the registration process for new surgical tools
[0038] 4. Example of execution through software
[0039] <<1. Overview of the present disclosure>>
[0040] The present disclosure can particularly update and use a surgical tool identifier after confirming that the surgical tool identifier is in a state that can fully recognize a new surgical tool.
[0041] First, refer to Figure 1 An overview of the present disclosure is described.
[0042] For a surgical tool identifier that identifies surgical tools that are tools used for surgery captured in a surgical image by a laparoscopic endoscope, etc., the present disclosure enables updating and using the surgical tool identifier after confirming that the surgical tool identifier has become in a state capable of recognizing a new surgical tool when the surgical tool identifier is updated to be capable of recognizing a new surgical tool.
[0043] Here, the learning process of the surgical tool identifier capable of registering and identifying a new surgical tool will be briefly described.
[0044] As a first step, Figure 1 As shown in the left part of , based on multiple intraperitoneal images P obtained by imaging a real image V of the new surgical tool with a laparoscope, etc. and changing the real image V to various angles, a three-dimensional computer graphics (3DCG) model M of the new surgical tool is generated by the structure from motion to be described below.
[0045] As a second step, Figure 1 As shown in the center part, multiple background images assumed to be captured by a laparoscope are generated, and the actual usage situation simulated by using the 3DCG model M of the new surgical tool is combined with the multiple background images to generate computer graphics (CG) data CGD1 to CGDn used as learning data.
[0046] As a third step, the surgical tool identifier R is learned by learning using computer graphics (CG) data CGD1 to CGDn.
[0047] Through the learning process as described above, the learned surgical tool identifier R identifies a new surgical tool from the intraperitoneal image P, and outputs a recognition result image RP obtained by superimposing and displaying the recognition result RI including the CG image on the real image V.
[0048] However, if the learning of the surgical tool identifier R is insufficient, the recognition position of the recognition result RI is unstable between frames, and the position of the recognition result RI appears to the user to change unstably relative to the real image V.
[0049] That is, for example, Figure 2 As shown in the corresponding recognition results RI, RI' and RI" in the frames Fa to Fc adjacent in the time direction from time ta to time tc, the stability of each position is low, and each position seems to change unsteadily and unstably relative to the position of the real image V.
[0050] In addition, if the 3DCG model M for the new surgical tool has a large deviation from the recognition result RI, that is, if the consistency between the 3DCG model M and the recognition result RI is low, the CG data generated using the current 3DCG model M cannot be used as appropriate learning data, and there is a possibility that the recognition accuracy cannot be effectively improved even if learning continues.
[0051] That is, the 3DCG model M on the right is Figure 3 When the recognition result RI shown on the left is thick, and the length Dr of the recognition result RI from the fulcrum portion to the distal end is different from the corresponding length Dm in the 3DCG model M, there is a possibility that the generated CG data cannot be used as appropriate learning data.
[0052] Therefore, in the present disclosure, when a surgical tool identifier that recognizes a surgical tool in a surgical image is updated to become capable of recognizing a new surgical tool, information indicating the recognition degree of the new surgical tool by the surgical tool identifier is presented to the user as a feedback image.
[0053] When it is determined that the recognition degree is low and the learning is insufficient based on the information indicating the recognition degree of the new surgical tool by the surgical tool identifier presented in the feedback image, the user causes imaging and learning of the new surgical tool to continue in the introduction environment.
[0054] Since information indicating the degree of recognition of new surgical tools by the surgical tool identifier is presented as a feedback image even while learning continues, the degree to which the recognition of new surgical tools by the surgical tool identifier improves as learning continues is presented to the user as needed.
[0055] Then, when the user considers that the recognition degree of the new surgical tool by the surgical tool identifier presented as the feedback image is sufficient, the user terminates the learning and updates and uses the surgical tool identifier.
[0056] Therefore, when the surgical tool identifier is updated so as to be able to recognize a new surgical tool, it is possible to prevent the occurrence of trouble due to the fact that the new surgical tool is not recognized during an actual surgery.
[0057] More specifically, the surgical tool identifier is constructed by learning based on learning data (computer graphics (CG) data) including a composite image generated by combining a 3DCG model simulating a new surgical tool to be identified with various intraperitoneal images.
[0058] Therefore, in the present disclosure, a feedback image including information for qualitatively and quantitatively evaluating the recognition degree of a new surgical tool by a surgical tool identifier is generated and presented to a user, thereby allowing the user to recognize the recognition degree of the new surgical tool.
[0059] The user determines whether the recognition degree of the new surgical tool by the surgical tool identifier is sufficient based on the information presented as the feedback image for qualitatively and quantitatively evaluating the recognition degree of the new surgical tool by the surgical tool identifier.
[0060] Here, when the user believes that the recognition degree of the surgical tool identifier for the new surgical tool is low and the learning is insufficient, the recognition degree of the surgical tool identifier for the new surgical tool is improved by continuing the learning using the learning data (CG data).
[0061] However, if the 3DCG model of a new surgical tool is not appropriately formed, even if learning data is randomly increased and learning is continued, there is a possibility that the recognition degree of the surgical tool identifier cannot be effectively improved.
[0062] Therefore, the 3DCG model of the current new surgical tool and the actual recognition result of the new surgical tool are presented to the user, so that the user can make a comparison and the user can qualitatively determine whether the 3DCG model of the current new surgical tool is appropriate.
[0063] Therefore, in a case where the user determines that the 3DCG model of the new surgical tool does not sufficiently match the recognition result of the new surgical tool, the recognition degree is improved by recreating the 3DCG model based on the current recognition result and then generating learning data (CG data) and continuing learning.
[0064] More specifically, by generating Figure 4 The feedback image shown is presented to the user, and the recognition degree of the new surgical tool by the surgical tool identifier is presented.
[0065] Figure 4 is a diagram for describing an overview of an appearance configuration example of a surgical image display device to which the technology of the present disclosure is applied.
[0066] Figure 4 The surgical image display device in the embodiment includes a display unit W and an operation unit C. The display unit W displays a feedback image presenting information indicating the recognition degree of the above-mentioned surgical tool identifier for a new surgical tool.
[0067] Show in Figure 4 The feedback image on the display unit W in the operation includes a field of view image display unit Wa during operation and a model display unit Wb. The field of view image display unit Wa during operation displays an operation image in the operation field captured by a laparoscopic endoscope not shown. The model display unit Wb displays a 3DCG model M for learning a surgical tool identifier.
[0068] exist Figure 4 In the intraoperative visual field image display unit Wa, a real image V of a surgical tool in a surgical image captured by a laparoscope or the like not shown and a recognition result RI of a surgical tool identifier are superimposed and displayed.
[0069] Since the gap between the real image V of the surgical tool and the recognition result RI of the surgical tool identifier is displayed by the display of the visual field image display unit Wa during surgery, the user can qualitatively recognize the recognition degree of the surgical tool identifier.
[0070] In addition, an evaluation display unit E is set in the field of view image display unit Wa during surgery, and displays an evaluation of the recognition result RI, for example, stability between frames relative to the recognition result RI and consistency between the 3DCG model M and the recognition result RI, that is, information for quantitatively evaluating the recognition degree of the surgical tool identifier.
[0071] By evaluating the display of the display unit E, the user can determine the recognition degree of the surgical tool identifier for a new surgical tool based on a quantitative evaluation.
[0072] also, Figure 4 The 3DCG model M of a new surgical tool used to learn the surgical tool identifier is displayed in the model display unit Wb.
[0073] Through the display of the model display unit Wb, the user can compare and view the real image V of the surgical tool, the recognition result RI and the 3DCG model M in the surgical field image display unit Wa, and thus can qualitatively identify the consistency between the current 3DCG model M and the recognition result RI of the actual surgical tool.
[0074] The operation unit C is provided with a learning continuation button B1 which is operated when continuing learning, a learning termination button B2 which is operated when terminating learning, and a 3DCG model re-creation button B3 which is operated when re-creating a 3DCG model.
[0075] When the user operates the operation unit C according to the display contents of the in-operation field image display unit Wa and the model display unit Wb in the feedback image displayed on the display unit W, continued learning of a new surgical tool identifier can be performed, learning can be terminated, or a 3DCG model can be recreated.
[0076] That is to say, Figure 4 In the in-operative visual field image display unit Wa, the real image V of the surgical tool and the recognition result RI are displayed in a shifted state, and a cross mark indicating that both stability and consistency in the evaluation display unit E are not appropriate is added.
[0077] In the case of such a display, the user can determine that the surgical tool identifier cannot sufficiently recognize the new surgical tool, that is, the learning of the new surgical tool identifier is insufficient, and thus operate, for example, the learning continuation button B1.
[0078] In addition, Figure 4 In the model display unit Wb, the 3DCG model M is expressed as being thicker than the real image V of the new surgical tool, and the lengths of the various parts of the 3DCG model M are also different from the recognition result RI. In the case of such a display, the user can determine that the consistency between the 3DCG model M of the new surgical tool and the recognition result RI is low, and therefore operate, for example, the 3DCG model re-creation button B3.
[0079] Therefore, the 3DCG model M is recreated based on the current recognition result RI, and thus the 3DCG model M is recreated in a state close to the recognition result RI, and CG data using the recreated 3DCG model M is created as learning data, and the learning of the surgical tool identifier is continued.
[0080] That is, in any case, since the feedback image is continuously displayed on the display unit W while the learning is continued, a state in which the recognition degree of the new surgical tool is improved by the continued learning is presented in the feedback image.
[0081] Then, for example, when the real image V of a new surgical tool and the recognition result RI are superimposed and displayed at a matching position, the 3DCG model M and the recognition result RI are displayed as having sufficient consistency, and based on the display contents of the in-operation field of view image display unit Wa and the model display unit Wb of the feedback image displayed on the display unit W, appropriate information indicating both stability and consistency in the evaluation display unit E is displayed, and it is considered that sufficient learning has been performed to reach a state where the new surgical tool identifier is able to recognize the new surgical tool.
[0082] In such a case, the user operates the learning termination button B2 to terminate the learning, and updates and uses the surgical tool identifier that has been learned for the new surgical tool.
[0083] In addition, in the present disclosure, when an evaluation based on a quantitative evaluation of the recognition result RI (for example, stability between frames relative to the recognition result RI and consistency between the 3DCG model M and the recognition result RI) is lower than a predetermined value, learning or recreating the 3DCG model M is continued without waiting for the user's operation.
[0084] Therefore, according to the present disclosure, in the case of updating the surgical tool identifier to be able to recognize a new surgical tool, the user can update and use the surgical tool identifier after confirming that the surgical tool identifier is able to fully recognize the new surgical tool.
[0085] <<2. Advantageous Implementation Method>>
[0086] Next, we will refer to Figure 5 A configuration example of an advantageous embodiment of a surgical image display system to which the technology of the present disclosure is applied is described.
[0087] Figure 5 The surgical image display system 11 in captures an image of a surgical field in the abdominal cavity, recognizes a surgical tool included in the captured image, and displays a CG image of the surgical tool as a recognition result to be superimposed on the surgical tool in the image.
[0088] The surgical image display system 11 includes a surgical tool identifier that recognizes a surgical tool included in an image through the image, generates a CG image representing the surgical tool recognized by the surgical tool identifier as a recognition result, and superimposes and displays the CG image on the surgical tool in the image.
[0089] For example, the surgical tool identifier is an identifier including a deep neural network (DNN) formed by machine learning using an image of a surgical field including surgical tools as learning data. Therefore, in order to enable the surgical tool identifier to recognize a new surgical tool, it is necessary to insert the new surgical tool into the abdominal cavity so that the RGB camera 41 of the laparoscope 31 captures an image, and perform learning processing based on the captured image.
[0090] In the learning process for causing the surgical tool identifier to recognize a new surgical tool, the surgical image display system 11 generates a feedback image indicating the degree of recognition of the new surgical tool by the surgical tool identifier, and presents the feedback image to the user.
[0091] The user confirms the recognition degree of the surgical tool identifier for the new surgical tool by viewing the feedback image presented by the surgical image display system 11, and continues learning when it is determined that the recognition degree is insufficient. Then, when it is determined that the learning is sufficient, the user terminates the learning of the surgical tool identifier, and updates and uses the surgical tool identifier.
[0092] More specifically, the surgical image display system 11 includes a laparoscope 31 and a surgical image display device 32 .
[0093] The laparoscope 31 includes an RGB camera 41 , captures an image of a surgical field including a surgical tool in the abdominal cavity, and outputs the captured image to the surgical image display device 32 as a surgical image.
[0094] For example, the RGB camera device 41 includes a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor, captures an image of a surgical field in the abdominal cavity serving as an imaging field, and outputs the image as image data to the surgical image display device 32.
[0095] When the learning process of the surgical tool identifier for a new surgical tool is performed, the laparoscope 31 images the new surgical tool from various angles by changing the new surgical tool to various angles, and outputs the captured images to the surgical image display device 32 .
[0096] When performing the learning process of the surgical tool identifier for a new surgical tool, the laparoscope 31 can capture images of the new surgical tool in an environment other than the surgical environment. However, in order to improve recognition accuracy in the surgical environment, it is desirable to capture images in the surgical environment.
[0097] The surgical image display device 32 includes a surgical tool identifier, identifies a surgical tool in the image based on the image provided from the laparoscope 31, and superimposes and displays a CG image representing the identified surgical tool on a real image of the surgical tool in the image.
[0098] The surgical image display device 32 performs a learning process (new surgical tool registration process) of a surgical tool identifier for recognizing a new surgical tool based on an image including the new surgical tool.
[0099] More specifically, the surgical image display device 32 includes a surgical image display processing unit 50 , a new surgical tool registration unit 51 , a learning data creation unit 52 , a storage device 53 , a display 54 , and an operation unit 55 .
[0100] The surgical image display processing unit 50 identifies the surgical tool based on the image of the surgical field including the surgical tool captured by the RGB camera device 41 of the laparoscope 31, generates an image obtained by superimposing a CG image representing the surgical tool as a recognition result on the surgical tool in the image, and displays the image on the display 54.
[0101] At this time, the surgical image display processing unit 50 reads the learned identifier 94 learned in a state capable of recognizing new surgical tools from the storage device 53, and recognizes the surgical tool based on the image of the surgical field including the surgical tool captured by the RGB camera 41 of the laparoscope 31.
[0102] That is, after registering the learned identifier 94 that is in a state of learning sufficient to be able to recognize a new surgical tool in the storage device 53, the surgical image display processing unit 50 performs surgical image display processing, recognizes the surgical tool, and displays on the display 54 a surgical image obtained by superimposing a CG image serving as a recognition result on the surgical tool.
[0103] More specifically, the surgery image display processing unit 50 includes a surgery tool identifier 61 and an image generating unit 62 .
[0104] The surgical tool identifier 61 includes an inference unit 61 a , reads the learned identifier 94 stored in the storage device 53 , identifies (infers) a surgical tool based on an image supplied from the laparoscope 31 , and supplies the recognition result to the image generation unit 62 .
[0105] The image generation unit 62 generates an image obtained by superimposing the CG image of the operation tool serving as the recognition result supplied from the operation tool identifier 61 on the operation tool in the image supplied from the laparoscope 31 , and displays the image on the display 54 .
[0106] When generating a new surgical tool identifier for registering a new surgical tool, the new surgical tool registration unit 51 generates a new surgical tool registration identifier 95 by learning based on the new surgical tool in the image provided from the laparoscope 31 and stores the new surgical tool registration identifier 95 in the storage device 53 .
[0107] More specifically, the new surgical tool registration unit 51 includes a 3DCG model generation unit 71 , a surgical tool identifier 72 , an evaluation unit 73 , and an image generation unit 74 .
[0108] The 3DCG model generation unit 71 generates a 3DCG model corresponding to a new surgical tool in the image based on the image supplied from the laparoscope 31 , and stores the 3DCG model in the storage device 53 as a 3DCG model 91 .
[0109] More specifically, the 3DCG model generation unit 71 includes a feature point matching processing unit 71a, which performs structure from motion processing by matching feature points in the area of the surgical tool between frames based on images of the new surgical tool captured at various angles provided from the laparoscope 31, generates three-dimensional point group information of the new surgical tool, and generates a 3DCG model of the new surgical tool using the generated three-dimensional point group information.
[0110] Note that the following will refer to Figure 7 and Figure 8 Details of a method of generating the 3DCG model 91 by the 3DCG model generating unit 71 are described.
[0111] The surgical tool identifier 72 has a configuration corresponding to the surgical tool identifier 61 of the surgical image display processing unit 50 and includes an inference unit 72 a. When the learning data creation unit 52 generates a new surgical tool registration identifier 95 based on the 3DCG model 91 , the surgical tool identifier 72 reads the new surgical tool registration identifier 95 .
[0112] Then, the surgical tool identifier 72 controls the inference unit 72a so that the new surgical tool registration identifier 95 is activated, identifies the new surgical tool based on the image provided from the laparoscope 31, generates a CG image corresponding to the new surgical tool as the identification result, and outputs the CG image to the evaluation unit 73 and the image generation unit 74.
[0113] The evaluation unit 73 generates information evaluating the recognition result based on the CG image and the 3DCG model 91 as the recognition result of the surgical tool identifier 72, and supplies the information to the image generation unit 74. Fig.11 and Fig.12 Describes the details of the evaluation of the recognition results.
[0114] The image generation unit 74 combines an image obtained by superimposing and displaying a CG image as a recognition result on a new surgical tool in an image provided from the laparoscope 31 with the display image of the 3DCG model 91, further generates a feedback image obtained by adding the evaluation result provided from the evaluation unit 73 to the superimposed image, and displays the feedback image on the display 54. Figure 6 Describe the feedback image.
[0115] When the user operates the operation unit 55 based on the feedback image displayed on the display 54 and instructs to recreate a 3DCG model for a new surgical tool, the 3DCG model generation unit 71 recreates the 3DCG model based on the current recognition result and stores it in the storage device 53 as a 3DCG model 91. Note that the following will refer to Fig.10 Describe the details of the recreation of the 3DCG model.
[0116] In addition, when the user operates the operation unit 55 based on the feedback image displayed on the display 54 and indicates the termination of learning, the new surgical tool registration unit 51 terminates the learning of the current new surgical tool registration identifier 95 and uses the new surgical tool registration identifier 95 to update the learned identifier 94 so that the surgical image display processing unit 50 can use the learned identifier 94.
[0117] When the new surgical tool registration unit 51 learns the new surgical tool registration identifier 95 and registers the new 3DCG model 91, the learning data creation unit 52 reads the new 3DCG model 91 and generates CG data used as learning data through simulation, and stores the generated CG data as new CG data 93 in the storage device 53.
[0118] Furthermore, the learning data creating unit 52 generates a new surgical tool registration identifier 95 by learning the learned identifier 94 using the new CG data 93 registered in the storage device 53 and the learned CG data 92 , and stores it in the storage device 53 .
[0119] More specifically, the learning data creating unit 52 includes a simulator 81 and a surgical tool identifier 82 .
[0120] The simulator 81 includes a 3DCG model reading unit 81 a and an image creating unit 81 b , and when a new 3DCG model 91 is registered in the storage device 53 , controls the 3DCG model reading unit 81 a to read the new 3DCG model 91 from the storage device 53 .
[0121] Then, the simulator 81 controls the image creation unit 81 b to generate various images of the surgical field, and generates new CG data 93 serving as learning data by performing CG synthesis on the generated images of the surgical field and the 3DCG model 91 , and stores it in the storage device 53 .
[0122] The surgical tool identifier 82 includes a learning processing unit 82 a , and controls the learning processing unit 82 a to generate a new surgical tool registration identifier 95 through a learning process using the learned CG data 92 , the new CG data 93 , and the learned identifier 94 stored in the storage device 53 .
[0123] Note that the following will refer to Figure 6 Details of a process in which the learning processing unit 82 a generates a new surgical tool registration identifier 95 through a learning process using the learned CG data 92 , the new CG data 93 , and the learned identifier 94 are described.
[0124] The display 54 is a display device including a liquid crystal display (LCD) or an organic electroluminescence (EL), and displays the surgical image supplied from the surgical image display processing unit 50 and the feedback image supplied from the new surgical tool registration unit 51 .
[0125] The operation unit 55 includes operation buttons, and when performing a learning process for registering a new surgical tool (new surgical tool registration process), the user's instructions based on the feedback image displayed on the display 54 are input to these buttons, and an operation signal corresponding to the operation of the corresponding button is provided to at least one of the new surgical tool registration unit 51 or the learning data creation unit 52.
[0126] More specifically, the operation unit 55 includes a 3DCG model re-creation button 55 a , a learning termination button 55 b , and a learning continuation button 55 c .
[0127] The 3DCG model re-creation button 55 a is a button operated when the user determines that re-creation of the 3DCG model is necessary based on the feedback image displayed on the display 54 .
[0128] When the 3DCG model re-creation button 55a is operated, the 3DCG model generation unit 71 re-creates the 3DCG model 91 based on the current recognition result. In addition, in the case where the consistency evaluation value obtained by the evaluation unit 73 is lower than a predetermined value, the 3DCG model generation unit 71 re-creates the 3DCG model 91 based on the current recognition result.
[0129] The learning termination button 55 b is a button operated when the user determines, based on the feedback image displayed on the display 54 , that the new surgical tool registration identifier 95 is able to sufficiently recognize the new surgical tool.
[0130] When the learning termination button 55b is operated, the surgical tool identifier 82 in the learning data creation unit 52 terminates the learning processing of the new surgical tool registration identifier 95 by the learning processing unit 82a, and uses the new surgical tool registration identifier 95 to update the learned identifier 94 so that the surgical image display processing unit 50 can use the learned identifier 94.
[0131] The learning continuation button 55 c is a button operated when the user determines based on the feedback image displayed on the display 54 that the recognition degree of the new surgical tool by the new surgical tool registration identifier 95 is low, considers that the learning is insufficient, and needs to continue the learning.
[0132] When the learning continuation button 55 c is operated, the surgical tool identifier 82 in the learning data creation unit 52 further continues the learning process of the new surgical tool registration identifier 95 by the learning processing unit 82 a using more CG data as learning data generated by the simulator 81 .
[0133] <Display example of feedback image>
[0134] Next, we will describe Figure 5 The appearance configuration example of the surgical image display device 32 is shown in FIG. Figure 6 Describes an example of displaying a feedback image.
[0135] Figure 6 Shows Figure 5 FIG. 3 is an example of the appearance configuration of the surgical image display device 32. Figure 6 In the embodiment, the display 54 is provided at the upper portion of the operation image display device 32 , and the operation unit 55 is provided at the lower portion of the operation image display device 32 .
[0136] Figure 6 An example of display of two feedback images showing the degree of recognition of the new surgical tool by the surgical tool identifier on the display 54 is shown.
[0137] Figure 6 The left part of shows a display example of a feedback image in a state where the recognition degree of the surgical tool identifier for a new surgical tool is insufficient, and Figure 6 The right part of shows an example of display of a feedback image in a state where the surgical tool identifier has sufficient recognition degree for a new surgical tool.
[0138] Figure 6 The feedback image displayed on the middle display 54 includes an image display unit 111 and a model display unit 112. The image display unit 111 displays an image captured by the laparoscope 31. The model display unit 112 displays a 3DCG model for learning a surgical tool identifier.
[0139] In the image display unit 111 , a real image 131 of a surgical tool in an image captured by the laparoscope 31 and a CG image 132 serving as a recognition result of a surgical tool identifier are superimposed and displayed.
[0140] Marks representing parts are attached to the CG image 132 including the recognition result, linear portions in the figure representing axes or frames are marked, and dots represent movable fulcrums or distal ends.
[0141] Through the display of the image display unit 111 , the gap between the real image 131 of the surgical tool and the CG image 132 including the recognition result of the surgical tool identifier is displayed.
[0142] That is to say, in the case of insufficient recognition, such as Figure 6 As shown in the left part of , the position of the CG image 132 including the recognition result offset from the real image 131 is expressed.
[0143] In contrast, when the recognition is sufficient, such as Figure 6 As shown in the right part of , a CG image 132 ′ including a recognition result matched with the real image 131 is expressed.
[0144] Through the display of the image display unit 111 , the user can qualitatively recognize the recognition degree by the surgical tool identifier.
[0145] In addition, the image display device 111 is provided with a stability evaluation display device 151 and a consistency evaluation display device 152, and respectively displays information indicating an evaluation of quantitative stability between frames of the CG image 132 as a recognition result, and information indicating an evaluation of quantitative consistency between the 3DCG models 141 and 141' and the CG image 132 as a recognition result, that is, information on a quantitative evaluation of the degree of recognition by the surgical tool identifier.
[0146] That is to say, in the case of insufficient recognition, such as Figure 6 As shown in the left part of , cross marks are attached to both the stability evaluation display unit 151 and the consistency evaluation display unit 152 , and it is insufficient to express both.
[0147] In contrast, when the recognition is sufficient, such as Figure 6 As shown in the right part of , check marks are attached to both the stability evaluation display unit 151 ′ and the consistency evaluation display unit 152 ′, and it is sufficient to express both.
[0148] Using the displays of the stability evaluation display units 151 and 151 ′ and the consistency evaluation display units 152 and 152 ′, the user can recognize the recognition degree of the surgical tool identifier for a new surgical tool based on the quantitative evaluation.
[0149] Note that the following will refer to Fig.11 and Fig.12 Details of the stability evaluation and the consistency evaluation displayed by the stability evaluation display units 151 and 151 ′ and the consistency evaluation display units 152 and 152 ′ are described.
[0150] In addition, 3DCG models 141 and 141' of new surgical tools for learning surgical tool identifiers are shown in FIG. Figure 6 In the model display unit 112.
[0151] Due to the display of the model display unit 112, the user can compare and view the real image 131 of the surgical tool, the CG images 132 and 132' as the recognition results, and the 3DCG models 141 and 141' on the image display unit 111, and thus can qualitatively identify the consistency between the 3DCG models 141 and 141', the current shape of the real image 131 of the actual surgical tool, and the CG images 132 and 132' as the recognition results.
[0152] That is, in cases where recognition is insufficient, e.g. Figure 6 As shown in the left part of , it indicates that the 3DCG model 141 is thicker than the real image 131 and different in shape from the real image 131, and the consistency is low.
[0153] In contrast, in cases where identification is sufficient, e.g. Figure 3 As shown in the right part of , it means that the shape of the 3DCG model 141 ′ matches the shape of the real image 131 and the consistency is high.
[0154] The operation unit 55 is provided with a 3DCG model re-creation button 55a which is operated when re-creating a 3DCG model, a learning termination button 55b which is operated when terminating learning, and a learning continuation button 55c which is operated when continuing learning.
[0155] When the user operates the operation unit 55 according to the display contents of the image display unit 111 and the model display unit 112 in the feedback image displayed on the display 54, it is possible to perform continued learning, terminate learning, or recreate a 3DCG model of a new surgical tool identifier.
[0156] That is to say, Figure 6 In the left part of , the real image 131 of the surgical tool and the CG image 132 as the recognition result are displayed in an offset state, and a cross mark is attached to both the stability evaluation display unit 151 and the consistency evaluation display unit 152 .
[0157] In the case of such a display, the user may determine that the new surgical tool registration identifier 95 cannot sufficiently recognize the new surgical tool, that is, the learning is insufficient, and thus operate, for example, the learning continuation button 55 c .
[0158] In addition, Figure 6In the left part of the model display unit 112, the 3DCG model 141 is expressed as being thicker than the real image 131 of the new surgical tool. In the case of such a display, the user can determine that the matching between the 3DCG model 141 of the new surgical tool and the real image 131 is insufficient, and therefore, for example, operates the 3DCG model re-creation button 55a.
[0159] Therefore, the 3DCG model 141 is recreated based on the current recognition result, and thus the 3DCG model 141 is recreated in a more accurate shape and size. Figure 3 As shown in the right part of , new learning data is created using the recreated 3DCG model 141 ′, and the learning of the surgical tool identifier is continued.
[0160] Furthermore, since the feedback image is continuously displayed on the display 54 while the learning is continued, the state in which the recognition degree of the new surgical tool is improved by the continued learning is presented in the feedback image.
[0161] Then, when the real image 131 of the new surgical tool and the CG image 132′ including the recognition result are superimposed and displayed in a matching state, and further, both the stability evaluation display unit 151 and the consistency evaluation display unit 152 are marked with a check mark to display information indicating appropriateness, and further, based on the display contents of the image display unit 111 and the model display unit 112 of the feedback image displayed on the display 54, the 3DCG model 141′, the CG image 132, and the real image 131 are displayed in shapes that are appropriately matched to each other, for example, as Figure 6 As shown in the right part of , it is considered that sufficient learning has been performed to reach a state where the new surgical tool registration identifier 95 is able to recognize new surgical tools.
[0162] In such a case, the user operates the learning termination button 55 b to terminate the learning, and the learned identifier 94 is updated and used with the new surgical tool registration identifier 95 for which learning is performed for the new surgical tool.
[0163] Therefore, in the case of updating the learned identifier 94 so as to be able to recognize a new surgical tool, after allowing the new surgical tool registration identifier 95 to learn enough to fully recognize the new surgical tool, the user can update and use the learned identifier 94 with the new surgical tool registration identifier 95.
[0164] <Method of generating 3DCG model>
[0165] Next, we will refer to Figure 7 and Figure 8 Describe the method of generating 3DCG models.
[0166] The 3DCG model generation unit 71 includes a feature point matching processing unit 71a, and performs structure from motion processing by matching feature points in the area of the surgical tool imaged from various angles continuously provided from the laparoscope 31 between frames, thereby generating three-dimensional point group information of a new surgical tool.
[0167] That is to say, the 3DCG model is Figure 7 As shown in the left part of FIG. 1 , in frame F1, a structure from motion process is performed on each feature point forming a feature point group 161 indicated by a black circle in the figure, and matching is continuously performed until frame Fn, forming a structure from motion process. Figure 7 The three-dimensional point group information 171 of the feature points is shown in the right part.
[0168] Furthermore, the 3DCG model generating unit 71 stores in advance labeling template models of 3DCG models for a plurality of existing surgical tools.
[0169] Therefore, the 3DCG model generation unit 71 searches for a template model that is closest to the generated three-dimensional point group information 171 among the template models of the stored 3DCG model of the existing surgical tool by matching the three-dimensional point group information 171 generated by the feature point matching processing unit 71a with the template model of the stored 3DCG model of the existing surgical tool.
[0170] Then, the 3DCG model generating unit 71 adjusts the size of the template model of the searched 3DCG model using the generated three-dimensional point group information 171 by fitting to generate a new 3DCG model of the surgical tool.
[0171] At this time, the 3DCG model generation unit 71 simply marks the mark attached to the searched template model in association with the 3DCG model of the new surgical tool.
[0172] That is, for example, the closest Figure 7 The template model of the 3DCG model of the shape of the three-dimensional point group information 171 generated in Figure 8 The situation of the template model 191 is shown in the left part.
[0173] In this case, the 3DCG model generation unit 71 adjusts the searched template model 191 by fitting according to the size of the generated three-dimensional point group information 171 to deform the template model 191 into Figure 8 The template model 191m is shown in the middle.
[0174] Then, the 3DCG model generating unit 71 sets the deformed template model 191m as the 3DCG model 91. At this time, the 3DCG model generating unit 71 simply marks the label 191a marked in advance in the template model 191 as the label 91a of the generated 3DCG model 91.
[0175] Notice, Figure 8 It is shown that the mark 191a indicating the movable fulcrum part for the template model 191 including the forceps is simply marked as the mark 91a of the 3DCG model 91. In addition, the marks may be set in a manner different from this example, and various marks for the distal end and the axis are used as simple marks for the generated 3DCG model 91.
[0176] <Learning>
[0177] Next, we will refer to Fig. 9 An example of learning a new surgical tool registration identifier 95 using the learned CG data 92 , the new CG data 93 , and the learned identifier 94 performed in the learning processing unit 82 a of the surgical tool identifier 82 in the learning data creating unit 52 is described.
[0178] The learning processing unit 82a of the surgical tool identifier 82 in the learning data creation unit 52 uses the learned CG data 92, the new CG data 93, and the learned identifier 94 registered in the storage device 53 by Fig. 9 The process shown enables new surgical tool registration identifier 95 to be learned.
[0179] That is, the learning processing unit 82a generates a new surgical tool registration identifier 95 (so that the new surgical tool registration identifier 95 is learned) by adding learning data including new CG data 93 containing the new surgical tool to learning data including learned CG data 92 that does not contain the new surgical tool, and initializing and learning the weights in the model of the learned identifier 94.
[0180] Note that when initializing the weights in the model of the learned identifier 94, the dimensionality of the learning processing unit 82a in the final layer is increased by the dimensionality associated with the new surgical tool.
[0181] Through such learning, the learning processing unit 82 a generates a new surgical tool registration identifier 95 that can recognize a new surgical tool in addition to the surgical tools that are identifiable in the learned identifier 94 .
[0182] Then, when the termination of learning is indicated, the learning processing unit 82a updates the learned identifier 94 with the new surgical tool registration identifier 95, and updates the learned CG data 92 with all the CG data used in the learning so far, that is, updates the learned CG data 92 by combining the information obtained from the original learned CG data 92 and the new CG data 93.
[0183] <Re-creation of 3DCG model>
[0184] Next, we will refer to Fig.10 Describe the recreation of a 3DCG model.
[0185] like Fig.10 As shown in the left part of , in a case where the three-dimensional point group information 171 of feature points includes a point group 171y and feature points 171x-1 to 171x-4 considered to be outliers, a template model of a 3DCG model having the closest shape is searched based on the distribution of the point group 171y and the feature points 171x-1 to 171x-4 considered to be outliers, and the size is adjusted.
[0186] So, for example, Figure 6 As shown in the 3DCG model 141 in the model display unit 112 of the feedback image in the left portion, it is conceivable that the 3DCG model 141 becomes thicker than the real image 131 and does not match the real image 131 .
[0187] In the recreation of the 3DCG model, the 3DCG model generation unit 71 re-searches the template model based only on the point 171y obtained by excluding the distribution of feature points 171x-1 to 171x-4 which are considered to be outliers and are not included within a predetermined distance from the frame of the CG image 132 which is the current recognition result, that is, the three-dimensional point group information as the search range is narrowed down based on the CG image 132 which is the recognition result, and the size is fitted with the point group 171y to recreate the 3DCG model generation unit.
[0188] Therefore, when recreating the 3DCG model, the distribution of the feature points 171x-1 to 171x-4 considered as outliers is excluded, so that the influence of the distribution of the feature points 171x-1 to 171x-4 considered as outliers is eliminated. Therefore, the 3DCG model is recreated so as to be close to the CG image 132 as the recognition result.
[0189] So, for example, Figure 6 As shown in the model display unit 112 of the feedback image in the right portion, a 3DCG model 141 ′ close to the real image 131 can be recreated.
[0190] <Stability Assessment>
[0191] Next, we will refer to Fig.11 Describe the stability assessment.
[0192] The stability evaluation is an index indicating how many CG images serving as the recognition result of the surgical tool identifier are continuously detected at stable positions.
[0193] That is, in a case where learning is performed sufficiently and the recognition degree of the surgical tool identifier is high, the position of the CG image serving as the recognition result does not change greatly and is constant to some extent, or does not change rapidly even if there is a change.
[0194] On the other hand, in a case where learning is insufficient and the recognition degree of the surgical tool identifier is low, the position of the CG image serving as the recognition result may greatly change, and rapid changes continue.
[0195] Therefore, the evaluation unit 73 obtains the average value or variance value of the magnitude of acceleration between consecutive frames at a predetermined position in the CG image serving as the recognition result as a quantitative stability evaluation value, and presents information corresponding to the stability evaluation value.
[0196] For example, Fig.11 As shown, in the image display unit 111, the average value or variance value of the amplitude of acceleration between frames of the CG image 132 used as the recognition result in each of the consecutive frames F11 to F13 at time t1 to time t3 is calculated in a time series, and is set as a stability evaluation value.
[0197] That is to say, Fig.11 As shown, a case where the CG image 132 at time t1 moves to a CG image 132' at time t2, and further moves to a CG image 132" at time t3 will be considered.
[0198] Here, in the CG image 132, four feature points such as feature points 132a to 132d are set, and corresponding feature points are set as feature points 132a' to 132d' in the CG image 132', and as feature points 132a" to 132d" in the CG image 132".
[0199] Here, for example, the evaluation unit 73 calculates the speed V1 between the feature point 132 a and the feature point 132 a ′ from the distance between the feature points and the time between frames.
[0200] Furthermore, for example, the evaluation unit 73 calculates the speed V2 between the feature point 132a′ and the feature point 132a″ from the distance between the feature points and the time between frames.
[0201] Then, the evaluation unit 73 calculates the acceleration of the feature point 132 a in the three frames from the speed difference ( V2 − V1 ) and the time in the frames.
[0202] Note that the method of obtaining the velocity and acceleration is just an example, and another method may be used.
[0203] The evaluation unit 73 similarly calculates the accelerations for feature points 132b to 132d, obtains the corresponding average values or variance values of the accelerations for feature points 132a to 132d, and sets the obtained values as stability evaluation values for evaluating stability, and sets the stability evaluation values as indicators indicating whether learning is insufficient or sufficient by comparing with a predetermined value.
[0204] That is, for example, in the case where the stability evaluation value of the average value or the variance value of the acceleration including the feature points 132b to 132d is greater than a predetermined value that can be regarded as having low stability, the evaluation unit 73 displays a cross mark to indicate insufficient learning on the stability evaluation display unit 151. In this case, the learning can be continued without waiting for the user's determination.
[0205] On the other hand, for example, in a case where the stability evaluation value of the average value or variance value of the acceleration including feature points 132b to 132d is less than a predetermined value that can be considered to have high stability, the evaluation unit 73 displays a check mark on the stability evaluation display unit 151 to indicate that the learning is sufficient.
[0206] Note that in the following description, a process in which learning is continued without waiting for user determination based on the comparison between the stability evaluation value and the predetermined value will be described. However, even in a state in which learning is considered sufficient based on the comparison between the stability evaluation value and the predetermined value and there is no need to continue learning, when the learning continuation button 55c is operated according to the user's determination, learning is continued.
[0207] <Consistency Assessment>
[0208] Next, we will refer to Fig.12 Describes conformance assessment.
[0209] Consistency evaluation is an indicator to evaluate the matching degree between the recognition result and the 3DCG model, that is, the consistency between the recognition result and the 3DCG model.
[0210] For example, even in the case where it is determined that the stability evaluation value is less than a predetermined value and learning is sufficient, if there is a deviation in shape, size, etc. between the recognition result and the 3DCG model, the consistency is low and it cannot be said that the recognition result is appropriate. In addition, in the case where there is a deviation in shape, size, etc. between the recognition result and the 3DCG model, it can be said that there is a deviation between the 3DCG model of the new surgical tool and the real image. That is, when the shape and size of the recognition result and the 3DCG model match each other, the deviation between the 3DCG model of the new surgical tool and the real image is also small.
[0211] Therefore, the evaluation unit 73 calculates a consistency evaluation value as an index indicating the degree to which the recognition result and the 3DCG model match each other through comparison between the recognition result and the 3DCG model.
[0212] More specifically, in the image display unit 111, the distance Dr (eg, Fig.12 ) and the distance Dm between the feature point 91x and the feature point 91y in the 3DCG model 91 (as shown in the left part of Fig.12 ), and quantitatively calculate the difference between the distance Dr and the distance Dm, the ratio between the distance Dr and the distance Dm, etc. to indicate the degree of coincidence between the two and set it as the consistency evaluation value.
[0213] Note that Fig.12 , both the CG image 132 and the 3DCG model 91 as the recognition result represent tweezers, the characteristic points 132a and 91x are the distal ends of the tweezers, and the characteristic points 132b and 91y are the movable fulcrum parts of the tweezers.
[0214] When the evaluation unit 73 compares the consistency evaluation value with the predetermined value and determines that the consistency evaluation value is greater than the predetermined value and the deviation between the recognition result and the 3DCG model is large, for example, a cross mark is displayed on the consistency evaluation display unit 152 to indicate that the 3DCG model needs to be recreated. In this case, the 3DCG model can be recreated without waiting for the user's determination.
[0215] On the other hand, when the evaluation unit 73 compares the consistency evaluation value with the predetermined value and determines that the consistency evaluation value is smaller than the predetermined value and the recognition result and the 3DCG model sufficiently match each other, for example, a check mark is displayed on the consistency evaluation display unit 152 .
[0216] Note that in the following description, the process of recreating the 3DCG model based on the comparison between the consistency evaluation value and the predetermined value without waiting for the user's determination will be described. However, even in a state where learning is considered sufficient based on the comparison between the consistency evaluation value and the predetermined value and the 3DCG model is not recreated, when the 3DCG model re-creation button 55a is operated according to the user's determination, the 3DCG model is recreated.
[0217] <New surgical tool registration process>
[0218] Next, we will refer to Fig.13 Flowchart description in Figure 5 New surgical tool registration processing is performed by the surgical image display device 32.
[0219] In step S31 , the new surgical tool registration unit 51 controls the display 54 to display an image that helps the laparoscope 31 capture images of the new surgical tool while changing the angle, and acquires an image in which the new surgical tool is captured while continuously changing the angle captured by the laparoscope 31 .
[0220] In step S32, the new surgical tool registration unit 51 controls the 3DCG model generation unit 71 to generate a 3DCG model based on the image in which the new surgical tool is captured. More specifically, the 3DCG model generation unit 71 controls the feature point matching processing unit 71a to detect feature points in the area in which the new surgical tool is captured from the captured image, and obtains three-dimensional point group information of the new surgical tool based on the continuous movement of the detected feature points.
[0221] In step S33 , the 3DCG model generating unit 71 searches for a template model that is closest to the three-dimensional point group information of the new surgical tool among the template models of the stored 3DCG models.
[0222] In step S34 , the 3DCG model generation unit 71 fits the template model of the 3DCG model closest to the three-dimensional point group information of the new surgical tool to the three-dimensional point group information, resizes it, and generates a 3DCG model corresponding to the new surgical tool.
[0223] In step S35 , the 3DCG model generation unit 71 simply marks the generated 3DCG model with a mark set as a template model searched when generating the 3DCG model, and stores the generated 3DCG model in the storage device 53 as the 3DCG model 91 of a new surgical tool.
[0224] In step S36, the learning data generating unit 52 controls the simulator 81 to generate CG data which is learning data corresponding to the 3DCG model 91 of the new surgical tool stored in the storage device 53. That is, the simulator 81 controls the 3DCG model reading unit 81a to read the 3DCG model 91 of the new surgical tool from the storage device 53.
[0225] In addition, the simulator 81 controls the image generation unit 81b to generate an image of the surgical field captured by the laparoscope 31, and combines the generated image with the read 3DCG model 91 of the new surgical tool to generate CG data used as learning data through simulation, and stores the generated CG data as new CG data 93.
[0226] In step S37, the surgical tool identifier 82 controls the learning processing unit 82a to generate a new surgical tool registration identifier 95 through learning. More specifically, the learning processing unit 82a reads the learned CG data 92, the new CG data 93, and the learned identifier 94 stored in the storage device 53, and generates a new surgical tool registration identifier 95 through learning. Fig. 9 The described method performs learning to generate a new surgical tool registration identifier 95 and stores it in the storage device 53.
[0227] In step S38, the new surgical tool registration unit 51 controls the surgical tool identifier 72 to identify the new surgical tool in the image captured by the laparoscope 31 by using the recognition processing of the new surgical tool registration identifier 95, and generates an image of the image display unit 111 in a feedback image, in which the recognition result is superimposed and displayed on the real image of the new surgical tool in the image.
[0228] In step S39, the evaluation unit 73 determines the Fig.11 and Fig.12 The described method calculates a stability evaluation value and a consistency evaluation value, compares each of the calculated stability evaluation value and consistency evaluation value with a predetermined value, and generates an evaluation result.
[0229] In step S40, the image generation unit 74 generates an image of the model display unit 112 based on the 3DCG model 91, adds the stability evaluation display unit 151 and the consistency evaluation display unit 152 to the image of the recognition result generated according to the evaluation result based on the stability evaluation value and the consistency evaluation value to generate the image display unit 111, and combines them to generate a feedback image, and displays the feedback image on the display 54.
[0230] In step S41 , the evaluation unit 73 determines that the stability evaluation value is equal to or less than the predetermined value, and in the case of determining that the stability evaluation value is not equal to or less than the predetermined value and the learning is insufficient, the process returns to step S36 .
[0231] That is, it is determined whether the stability evaluation value is equal to or less than a predetermined value, and in a case where it is determined that the stability evaluation value is not equal to or less than the predetermined value and learning is insufficient, learning is continued through the processing of step S36 to step S41.
[0232] Then, in step S41 , in the case where it is determined that the stability evaluation value is equal to or smaller than the predetermined value and the learning is sufficient, the process proceeds to step S42 .
[0233] In step S42 , the evaluation unit 73 determines whether the consistency evaluation value is equal to or less than a predetermined value, and in the case where it is determined that the consistency evaluation value is not equal to or less than the predetermined value and the 3DCG model needs to be recreated, the process proceeds to step S46 .
[0234] In step S46, as shown in FIG. Fig.10 As described, the 3DCG model generating unit 71 narrows down the search range of the three-dimensional point group information based on the recognition result.
[0235] Then, the process returns to step S33 , and the 3DCG model generation unit 71 recreates the 3DCG model based on the three-dimensional point group information in which the search range is narrowed down based on the recognition result, and continues the learning.
[0236] That is, in step S41, in the case where it is determined that the 3DCG model needs to be recreated, step S46 and the processing in steps S33 to S41 are repeated to recreate the 3DCG model, and learning using the recreated 3DCG model is repeated.
[0237] Then, in step S41 , in the case where it is determined that the consistency evaluation value is equal to or smaller than the predetermined value and there is no need to recreate the 3DCG model, the process proceeds to step S43 .
[0238] In step S43 , the new operation tool registration unit 51 determines whether the learning termination button 55 b has been operated, and in a case where the learning termination button has not been operated, the process proceeds to step S44 .
[0239] In step S44 , the new operation tool registration unit 51 determines whether the learning continuation button 55 c has been operated, and in a case where the learning continuation button has not been operated, the process proceeds to step S45 .
[0240] In step S45 , the learning data creating unit 52 determines whether the 3DCG model re-creation button 55 a has been operated, and in the case where the 3DCG model re-creation button has not been operated, the process returns to step S41 .
[0241] That is, when the 3DCG model re-creation button 55a, the learning termination button 55b and the learning continuation button 55c of the operation unit 55 are not operated by the processing in steps S43 to S45, the processing in steps S41 to S45 is continued, and the state of displaying the feedback image is continued.
[0242] In step S45, in the case where the user has operated the 3DCG model re-creation button 55a based on the feedback image, the process proceeds to step S46, the 3DCG model is re-created, and learning based on the re-created 3DCG model is continued.
[0243] Furthermore, in step S44 , in the case where the user has operated the learning continuation button 55 c based on the feedback image, the process returns to step S36 , and the learning is continued.
[0244] Then, in step S43 , in the case where the user has operated the learning end button 55 b based on the feedback image, the process proceeds to step S47 .
[0245] In step S47 , the new surgical tool registration unit 51 updates the learned identifier 94 with the generated new surgical tool registration identifier 95 , and adds the new CG data 93 to the learned CG data 92 to update the learned CG data 92 , and terminates the processing.
[0246] That is, through the above series of processing, a 3DCG model 91 of a new surgical tool is generated by capturing an image of the new surgical tool while changing the angle, new CG data 93 as learning data is generated using the generated 3DCG model, a new surgical tool registration identifier 95 is generated based on learning using the generated new CG data 93, the learned CG data 92 and the learned identifier 94, the new surgical tool is identified by the generated new surgical tool registration identifier 95, an evaluation value based on the identification result is calculated, and learning is continued or the 3DCG model is recreated, and learning is continued based on the evaluation result.
[0247] At this time, since the 3DCG model, the recognition result, and the evaluation value are displayed as the feedback image, the user can continue to learn or recreate the 3DCG model by himself based on the feedback image based on the 3DCG model, the recognition result, and the evaluation value.
[0248] Therefore, when registering a new surgical tool in a recognizable state, the user causes the laparoscope 31 to capture an image of the new surgical tool while changing the angle in the surgical environment, and performs the above-mentioned series of new surgical tool registration processes, and updates the learned identifier 94 after confirming that the new surgical tool is sufficiently recognizable, and uses the learned identifier 94 in the actual operation in the surgical image display processing unit 50.
[0249] Note that, in the above, an example has been described in which a new surgical tool registration process is performed by the new surgical tool registration unit 51, the learning data creation unit 52, the storage device 53, the display 54, and the operation unit 55 in the surgical image display device 32. However, each of the new surgical tool registration unit 51, the learning data creation unit 52, and the storage device 53 may be implemented by a server computer or a cloud computer on a network.
[0250] <<3. Modification of the registration process for new surgical tools>>
[0251] An example has been described above in which a 3DCG model is generated by capturing an image of a new surgical tool while changing an angle, and then CG data as learning data is generated before starting surgery so that the new surgical tool registration recognizer 95 performs learning.
[0252] However, in the case where learning is felt to be insufficient after starting operation after a new surgical tool is registered once, the process of generating a 3DCG model may be omitted because a 3DCG model corresponding to the new surgical tool has already been generated.
[0253] Therefore, reference will be made to Fig.14 The flowchart of describes a modification of a new surgical tool registration process in which the process of generating a 3DCG model is omitted when the 3DCG model has already been generated.
[0254] Notice, Fig.14 The processing of steps S62 to S78 in the flowchart of Fig.13 The processing of steps S31 to S47 in is similar, and therefore the description thereof will be omitted.
[0255] That is, in step S61 , the new surgery tool registration unit 51 determines whether the 3DCG model 91 has been registered in the storage device 53 .
[0256] In step S61 , in the case where it is determined that the 3DCG model 91 is not registered in the storage device 53 , the process proceeds to step S62 , and an image of a new surgical tool is captured while changing the angle, and a 3DCG model 91 corresponding to the new surgical tool is generated.
[0257] On the other hand, in step S61 , in the event that determination is made that the 3DCG model 91 has already been registered in the storage device 53 , the processing in steps S62 to S66 is skipped, and learning of a new surgical tool registration identifier 95 using the 3DCG model is started.
[0258] According to the above process, when it is desired to continue learning or recreate a 3DCG model again after performing the process of registering a new surgical tool once, it is possible to omit the process of capturing an image of a new surgical tool and creating the 3DCG model 91. Therefore, it is possible to more easily continue learning of a new surgical tool and improve recognition accuracy.
[0259] Furthermore, at this time, even in the case of recreating the 3DCG model 91, a 3DCG model 91 more similar to the new surgical tool can be generated after narrowing down the three-dimensional point group information using the recognition result, and higher-precision learning can be achieved.
[0260] <<4. Example of execution by software>>
[0261] Incidentally, the above series of processing can be performed by hardware, but can also be performed by software. In the case of performing a series of processing by software, the program constituting the software is installed from a recording medium to, for example, a computer built into dedicated hardware or a general-purpose computer capable of performing various functions by installing various programs, etc.
[0262] Fig.15 1002 and 1003. The computer includes a central processing unit (CPU) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A read only memory (ROM) 1002 and a random access memory (RAM) 1003 are connected to the bus 1004.
[0263] An input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009 are connected to the input / output interface 1005. The input unit 1006 includes an input device through which a user inputs an operation command, such as a keyboard and a mouse, the output unit 1007 outputs a processing operation screen and an image of a processing result to a display device, the storage unit 1008 includes a hard disk drive and the like and stores programs and various data, and the communication unit 1009 includes a local area network (LAN) adapter and the like and performs communication processing via a network represented by the Internet. In addition, a drive 1010 that reads and writes data from and to a removable storage medium 1011, such as a magnetic disk (including a floppy disk), an optical disk (including a compact disc-read only memory (CD-ROM) and a digital versatile disc (DVD)), a magneto-optical disk (including a mini disc (MD)), or a semiconductor memory is connected.
[0264] The CPU 1001 executes various processes according to a program stored in the ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 into the RAM 1003. In addition, the RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes and the like.
[0265] In the computer configured as described above, for example, the CPU 1001 loads a program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program to thereby perform the above-described series of processing.
[0266] The program executed by the computer (CPU 1001) can be provided by being recorded in, for example, the removable storage medium 1011 as a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0267] In the computer, by attaching a removable storage medium 1011 to a drive 1010, the program can be installed in the storage unit 1008 via the input / output interface 1005. In addition, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be pre-installed in the ROM 1002 or the storage unit 1008.
[0268] Note that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or a program that executes processing in parallel or at necessary timing (for example, when a call is made).
[0269] Notice, Fig.15 CPU 1001 implementation in Figure 5 The storage unit 1008 implements the functions of the new surgical tool registration unit 51 and the learning data creation unit 52. Figure 5 The function of the storage unit 53 in the input unit 1006 is realized Figure 5 The function of the operating unit 55 in.
[0270] In addition, in this specification, a system is intended to mean an assembly of multiple components (devices, modules (components), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a device in which multiple modules are housed in one housing are both systems.
[0271] Note that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present disclosure.
[0272] For example, the present disclosure may have a configuration of cloud computing in which one function is shared by a plurality of computers via a network and processing is performed collaboratively.
[0273] Furthermore, each of the steps described in the above flowcharts may be performed by one device, or may be performed by a plurality of devices in a shared manner.
[0274] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be executed by a plurality of devices in a shared manner.
[0275] Note that the present disclosure may have the following configurations.
[0276] <1> An information processing device, comprising:
[0277] a CG (Computer Graphics) data generating unit configured to generate CG data based on an image including a new surgical tool, the CG data being used as learning data for learning of an identifier for recognizing the new surgical tool;
[0278] a learning unit configured to cause an identifier for identifying the new surgical tool to learn based on the CG data; and
[0279] an evaluation unit configured to calculate an evaluation value that evaluates a recognition result of the new surgical tool by the identifier based on the image, wherein
[0280] The learning unit continues the learning of the identifier based on the evaluation value.
[0281] <2> According to the claim <1> The information processing device, wherein
[0282] The evaluation unit calculates an inter-frame acceleration of a recognition position of the recognition result of the new surgical tool by the identifier as a stability evaluation value, and
[0283] The learning unit continues the learning of the identifier according to the stability evaluation value.
[0284] <3> According to the claim <2> The information processing device, wherein
[0285] When the stability evaluation value is greater than a predetermined value, the learning unit continues the learning of the identifier.
[0286] <4> According to the claim <1> The information processing device further includes:
[0287] A display unit configured to generate and display a feedback image in which a CG image corresponding to the recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image.
[0288] <5> According to the claim <4> The information processing device, wherein
[0289] In the feedback image, the CG (computer graphics) image corresponding to the recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image, and information indicating whether to continue the learning of the identifier is displayed according to the evaluation value.
[0290] <6> According to the claim <5> The information processing device further includes:
[0291] a learning continuation instruction operation unit, which is operated by the user to instruct to continue the learning of the identifier, wherein:
[0292] When the learning continuation instruction operation unit is operated, the learning unit continues the learning of the identifier.
[0293] <7> According to the claim <5> The information processing device further includes:
[0294] a learning termination instruction operation unit, which is operated by a user when instructing to terminate the learning of the identifier, wherein:
[0295] When the learning termination instruction operation unit is operated, the learning unit terminates the learning of the identifier.
[0296] <8> According to the claim <1> The information processing device further includes:
[0297] A 3DCG (three-dimensional computer graphics) model generation unit is configured to generate a 3DCG model of the new surgical tool based on the image, wherein:
[0298] The CG data generating unit generates the CG data used as learning data for learning of an identifier for recognizing the new surgical tool using the 3DCG model.
[0299] <9> According to the claim <8> The information processing device, wherein
[0300] The 3DCG model generation unit generates three-dimensional point group information of the new surgical tool based on the image, searches for a template model closest to the three-dimensional point group information among the template models of multiple pre-stored 3DCG models of surgical tools, and fits the searched template model based on the three-dimensional point group information to generate the 3DCG model.
[0301] <10> According to the claim <9> The information processing device, wherein
[0302] The evaluation unit calculates a difference between the 3DCG model and a recognition result of the new surgical tool by the identifier as a consistency evaluation value, and
[0303] The 3DCG model generation unit recreates the 3DCG model according to the consistency evaluation value.
[0304] <11> According to the claim <10> The information processing device, wherein
[0305] When the consistency evaluation value is greater than a predetermined value, the 3DCG model generation unit recreates the 3DCG model.
[0306] <12> According to the claim <10> The information processing device, wherein
[0307] When recreating the 3DCG model, the 3DCG model generation unit recreates the 3DCG model using the three-dimensional point group information narrowed down based on the recognition result of the new surgical tool, among the three-dimensional point group information used when creating the 3DCG model.
[0308] <13> According to the claim <8> The information processing device further includes:
[0309] A display unit configured to generate and display a feedback image in which a CG image corresponding to a recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image, and the 3DCG model is displayed.
[0310] <14> According to the claim <13> The information processing device, wherein
[0311] in,
[0312] The evaluation unit calculates a difference between the 3DCG model and a recognition result of the new surgical tool by the identifier as a consistency evaluation value, and
[0313] In the feedback image, information indicating whether the 3DCG model is to be recreated is displayed according to the consistency evaluation value.
[0314] <15> According to the claim <14> The information processing device further includes:
[0315] A re-creation instruction operation unit is operated by a user when instructing to re-create the 3DCG model, wherein:
[0316] When the re-creation instruction operation unit is operated, the 3DCG model generation unit re-creates the 3DCG model.
[0317] <16> According to the claim <1> to <15> The information processing device as described in any one of the preceding claims, wherein:
[0318] The images are images captured by a laparoscopy.
[0319] <17> An information processing method comprises the following steps:
[0320] generating CG (computer graphics) data based on an image including a new surgical tool, the CG data being used as learning data for learning of a recognizer for recognizing the new surgical tool;
[0321] causing the recognizer for recognizing the new surgical tool to learn based on the CG data; and
[0322] The evaluation value is calculated based on the recognition result of the new surgical tool by the identifier based on the image, wherein
[0323] The learning of the identifier is continued according to the evaluation value.
[0324] <18> A program for causing a computer to:
[0325] a CG (Computer Graphics) data generating unit configured to generate CG data based on an image including a new surgical tool, the CG data being used as learning data for learning of an identifier for recognizing the new surgical tool;
[0326] a learning unit configured to learn the recognizer for recognizing the new surgical tool based on the CG data; and
[0327] an evaluation unit configured to calculate an evaluation value that evaluates a recognition result of the new surgical tool by the identifier based on the image, wherein
[0328] The learning unit continues the learning of the identifier based on the evaluation value.
[0329] Reference numerals list
[0330] 11. Surgical Image Display System
[0331] 31 Laparoendoscopy
[0332] 32. Surgical image display device
[0333] 41 RGB camera device
[0334] 50. Surgical image display processing unit
[0335] 51 New surgical tool registration unit
[0336] 52 Learning Data Creation Unit
[0337] 53 Storage device
[0338] 54 Display
[0339] 55 Operating unit
[0340] 55a 3DCG model creation button
[0341] 55b Learning end button
[0342] 55c Learning Continue Button
[0343] 61 Surgical tool identifier
[0344] 61a Inference Unit
[0345] 62 Image generation unit
[0346] 71 3DCG model generation unit
[0347] 71a Feature point matching processing unit
[0348] 72 Surgical tool identifier
[0349] 72a Inference Unit
[0350] 73 evaluation units
[0351] 74 Image generation unit
[0352] 81 Simulator
[0353] 81a 3DCG model reading unit
[0354] 81b Image creation unit
[0355] 82 Surgical tool identifier
[0356] 82a Learning Processing Unit
[0357] 91 3DCG Models
[0358] 92 Learned CG data
[0359] 93 New CG Data
[0360] 94 Learned Recognizer
[0361] 95 New surgical tool registration identifier
Claims
1. An information processing device, comprising: a computer graphics (CG) data generating unit configured to generate CG data based on an image including a new surgical tool, the CG data being used as learning data for learning of an identifier for recognizing the new surgical tool; a learning unit configured to enable an identifier for identifying the new surgical tool to learn based on the CG data; as well as an evaluation unit configured to calculate an evaluation value that evaluates a recognition result of the new surgical tool by the identifier based on the image, wherein The learning unit continues the learning of the identifier based on the evaluation value.
2. The information processing device according to claim 1, wherein: The evaluation unit calculates an inter-frame acceleration of a recognition position of a recognition result of the new surgical tool by the identifier as a stability evaluation value, and The learning unit continues the learning of the identifier according to the stability evaluation value.
3. The information processing device according to claim 2, wherein: When the stability evaluation value is greater than a predetermined value, the learning unit continues the learning of the identifier.
4. The information processing device according to claim 1, further comprising: A display unit configured to generate and display a feedback image in which a CG image corresponding to a recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image.
5. The information processing device according to claim 4, wherein: In the feedback image, the CG (computer graphics) image corresponding to the recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image, and information indicating whether to continue the learning of the identifier is displayed according to the evaluation value.
6. The information processing device according to claim 5, further comprising: a learning continuation instruction operation unit, which is operated by the user to instruct to continue the learning of the identifier, wherein: When the learning continuation instruction operation unit is operated, the learning unit continues the learning of the identifier.
7. The information processing device according to claim 5, further comprising: a learning termination instruction operation unit, which is operated by a user when instructing to terminate the learning of the identifier, wherein: When the learning termination instruction operation unit is operated, the learning unit terminates the learning of the identifier.
8. The information processing device according to claim 1, further comprising: A 3DCG (three-dimensional computer graphics) model generation unit is configured to generate a 3DCG model of the new surgical tool based on the image, wherein: The CG data generating unit generates the CG data used as learning data for learning of an identifier for recognizing the new surgical tool using the 3DCG model.
9. The information processing device according to claim 8, wherein: The 3DCG model generation unit generates three-dimensional point group information of the new surgical tool based on the image, searches for a template model closest to the three-dimensional point group information among the template models of multiple pre-stored 3DCG models of surgical tools, and fits the searched template model based on the three-dimensional point group information to generate the 3DCG model.
10. The information processing device according to claim 9, wherein: The evaluation unit calculates a difference between the 3DCG model and a recognition result of the new surgical tool by the identifier as a consistency evaluation value, and The 3DCG model generation unit recreates the 3DCG model according to the consistency evaluation value.
11. The information processing device according to claim 10, wherein: When the consistency evaluation value is greater than a predetermined value, the 3DCG model generation unit recreates the 3DCG model.
12. The information processing device according to claim 10, wherein: When recreating the 3DCG model, the 3DCG model generation unit recreates the 3DCG model using the three-dimensional point group information narrowed down based on the recognition result of the new surgical tool, among the three-dimensional point group information used when creating the 3DCG model.
13. The information processing device according to claim 8, further comprising: A display unit configured to generate and display a feedback image in which a CG image corresponding to a recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image and the 3DCG model is displayed.
14. The information processing device according to claim 13, wherein: In the feedback image, a computer graphics (CG) image corresponding to the recognition result of the new surgical tool is superimposed and displayed on the new surgical tool in the image, and the 3DCG model is displayed, and information indicating the necessity of recreating the 3DCG model is displayed according to the evaluation value.
15. The information processing device according to claim 14, further comprising: A re-creation instruction operation unit is operated by a user when instructing to re-create the 3DCG model, wherein: When the re-creation instruction operation unit is operated, the 3DCG model generation unit re-creates the 3DCG model.
16. The information processing device according to claim 1, wherein: The images are images captured by a laparoscopy.
17. An information processing method comprising the following steps: generating CG (computer graphics) data based on an image including a new surgical tool, the CG data being used as learning data for learning of a recognizer for recognizing the new surgical tool; causing the recognizer for recognizing the new surgical tool to learn based on the CG data; as well as An evaluation value is calculated, wherein the evaluation value evaluates the recognition result of the new surgical tool by the recognizer based on the image, wherein The learning of the identifier is continued according to the evaluation value.
18. A program for causing a computer to: a CG (Computer Graphics) data generating unit configured to generate CG data based on an image including a new surgical tool, the CG data being used as learning data for learning of an identifier for recognizing the new surgical tool; a learning unit configured to learn the recognizer for recognizing the new surgical tool based on the CG data; as well as an evaluation unit configured to calculate an evaluation value that evaluates a recognition result of the new surgical tool by the identifier based on the image, wherein The learning unit continues the learning of the identifier based on the evaluation value.
