Surgical image processing platform and computer program
The surgical image processing platform empowers users to customize inference models, drawing modes, and display settings, addressing the limitations of predetermined configurations in existing systems by enhancing flexibility and accuracy in surgical image analysis and display.
Patent Information
- Application Number
- JP2024145877
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-04-11
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing surgical support systems lack user flexibility in selecting and combining trained models, surgical support information content, and display modes for surgical images, as these are predetermined by the system provider.
A surgical image processing platform that allows users to individually set inference models, drawing modes, and display modes, with multiple models and modes available for each layer, and supports input and output ports for data flow customization.
Enables users to analyze, draw, and display surgical images uniquely, enhancing flexibility and accuracy by allowing multiple model and mode combinations, improving usability and analysis precision.
Smart Images

Figure 2024167305000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a surgical image processing platform and computer program for processing surgical images captured during a surgery performed by a surgeon. [Background technology]
[0002] BACKGROUND ART Conventionally, a technique for assisting surgery is known in which surgical images are acquired by capturing images of a surgery performed by a surgeon, and the surgical images are analyzed and the analysis results are applied to the surgical images.
[0003] For example, Patent Document 1 proposes a surgical support system having an analysis unit that generates risk analysis information for a first surgical image by applying the first surgical image acquired by an acquisition unit to a trained model generated using training data that includes a second surgical image, which is a surgical image different from the first surgical image, and information regarding the risk of complications due to surgery, and an output unit that superimposes surgical support information based on the risk analysis information generated by the analysis unit on the surgical image and outputs it. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent Publication No. 2021-29258 Summary of the Invention [Problem to be solved by the invention]
[0005] Incidentally, the trained model used in the surgery assistance system may be generated using training data accumulated for each user. Furthermore, the surgical assistance information generated by the trained model may differ in terms of the content of information required by each user. Furthermore, the display mode in which the surgical assistance information is superimposed on the surgical image may differ in terms of ease of viewing and the support provided for the surgery for each user. For this reason, users of the surgery assistance system desire the ability to freely select and combine the trained model, the content of the surgical assistance information, and the display mode in which the surgical assistance information is superimposed on the surgical image.
[0006] However, in the surgical support system of Patent Document 1, the trained model, the content of the surgical support information, and the display format in which the surgical support information is superimposed on the surgical image are set by the provider of the surgical support system, and cannot be freely selected by the user of the surgical support system.
[0007] The present invention has been made in consideration of the above points, and aims to provide a surgical image processing platform and computer program that allows for the free setting of models to be implemented in layers that perform analysis, drawing, and display processes in the processing of surgical images. [Means for solving the problem]
[0008] (1) A surgical image processing platform for processing surgical images captured during a surgery performed by a surgeon, an inference means for setting an inference model for analyzing the surgical image; a calculation means for setting a drawing mode to generate a drawing image in which the analysis result of the inference model is reflected in the surgical image; a display setting means for setting a display mode in which the drawn image is displayed on a display means in a predetermined mode, A surgical image processing platform characterized in that the inference model, the drawing manner, and the display manner are each set individually.
[0009] In the invention (1), the surgical image processing platform comprises an inference means, a calculation means, and a display setting means, and processes surgical images captured during surgery performed by a surgeon. The inference means is configured with an inference model that analyzes surgical images. The calculation means is set with a drawing mode for generating a drawing image that reflects the analysis results of the inference model on the surgical image. The display setting means sets a display mode for displaying the drawn image on the display means in a predetermined mode. The surgical image processing platform has an inference model, a rendering mode, and a display mode that are individually set.
[0010] According to the invention (1), an inference means capable of setting an inference model, a calculation means capable of setting a drawing mode, and a display setting means capable of setting a display mode are provided. The inference model, drawing mode, and display mode can be set individually.
[0011] This allows users of the surgical image processing platform to, for example, individually set their own inference model, drawing style, and display style, thereby analyzing surgical images in their own way, generating drawing images in their own way based on the analysis results, and displaying these drawing images in their own way. Therefore, it is possible to provide a surgical image processing platform that allows for the free setting of models to be implemented in layers that perform analysis, drawing, and display processing in the processing of surgical images.
[0012] (2) The inference means can set a plurality of the inference models, The calculation means can set a plurality of the drawing modes, The surgical image processing platform according to (1), wherein the display setting means is capable of setting a plurality of the display modes.
[0013] According to the invention of (2), multiple inference models can be set in the inference means, multiple drawing modes can be set in the calculation means, and multiple display modes can be set in the display setting means. This makes it possible to equip each layer (inference means, calculation means, display setting means) in the processing of surgical images with various functions (functions realized by each model).
[0014] (3) A surgical image processing platform as described in (1) or (2), characterized in that the inference means, the calculation means, and the display setting means each have an input port for inputting data and an output port for outputting data that are individually set.
[0015] According to the invention of (3), the input port for inputting data and the output port for outputting data can be set individually in the inference means, the calculation means, and the display setting means, so that the data input source and the data output destination can be set individually in each layer (inference means, calculation means, display setting means) in the processing of surgical images. This improves the flexibility of the input source and output destination of each layer.
[0016] (4) The surgical image processing platform described in (3) is characterized in that the inference means, the calculation means, and the display setting means are each capable of setting a plurality of the input ports and a plurality of the output ports.
[0017] According to the invention (4), the inference means, the calculation means, and the display setting means can each be set to a plurality of input ports and a plurality of output ports, which makes it possible to input various data and output data in various directions (for example, to another device or another layer) at each layer (the inference means, the calculation means, the display setting means) in the processing of surgical images.
[0018] (5) The input port of the inference means An external device can be connected to the output port of the display setting means, The data output from the external device and the data output from the display setting means can be input, The input port of the computing means is connectable to the output port of the inference means and / or the display setting means, data output from the inference means and / or the display setting means can be input; The input port of the display setting means is connectable to the output port of the computing means; The surgical image processing platform according to (3) or (4), characterized in that data output from the calculation means can be input.
[0019] According to the invention of (5), in addition to data from an external device, data from the display setting means can be input to the inference means. Also, in addition to data from the inference means, data from the display setting means can be input to the calculation means. This makes it possible to feed back data from a means located downstream in the process (e.g., the inference means or the display setting means for the inference means) to a means located upstream in the process (e.g., the inference means or the calculation means for the display setting means). This makes it possible to obtain multiple types of results, for example, by repeatedly using data input from an external device and processing it with different inference models or rendering modes.
[0020] (6) The surgical image processing device further includes a preprocessing means for converting the image quality of the surgical image. The surgical image processing platform described in (1) is characterized in that the inference means is configured to set an inference model for analyzing the surgical image whose image quality has been converted.
[0021] There are various types of external devices (e.g., endoscope systems, endoscopes, etc.) that capture images of a surgical procedure, and the quality of the captured images varies depending on the device. Depending on the quality of the surgical images, the accuracy of the analysis results may be reduced.
[0022] According to the invention (6), the preprocessing means converts the image quality of the surgical images obtained by capturing the surgery, and the inference means analyzes the surgical images whose image quality has been converted.
[0023] This allows the surgical image processing platform to convert the image quality of the surgical image, for example, to an image quality that improves the analytical accuracy of the inference means, rather than simply analyzing the surgical image input as is, and then analyze the surgical image with converted image quality, thereby preventing a decrease in the accuracy of the analysis results.
[0024] (7) The surgical image processing platform according to (6), wherein the preprocessing means is capable of outputting a converter that converts the image quality of the surgical image.
[0025] According to the invention (7), the preprocessing means can output a converter that converts the image quality of the surgical image.
[0026] With this configuration, it is possible to provide a converter for a surgical image processing platform used by a user to, for example, the provider of the surgical image processing platform or another user. This makes it possible to improve or reuse the converter that converts the image quality of the surgical image, thereby improving usability.
[0027] (8) The surgical image processing platform described in (6) or (7) is characterized in that the preprocessing means converts the image quality of the surgical image using a conversion formula corresponding to the image quality of the image learned by the inference model.
[0028] If the external device that captured the surgical images on which the inference model learned is different from the external device that acquires the surgical images to be analyzed by the inference model, the image quality of the surgical images captured by these external devices will also be different, which may reduce the accuracy of the analysis results.
[0029] According to the invention (8), the preprocessing means converts the image quality of the surgical image taken of the surgery to be analyzed using a conversion formula corresponding to the image quality of the image learned by the inference model, thereby preventing a decrease in the accuracy of the analysis results.
[0030] (9) A surgical image processing platform for processing surgical images captured by a surgeon during a surgery, an inference means for setting an inference model for analyzing the surgical image; a calculation means for setting a drawing mode to generate a drawing image in which the analysis result of the inference model is reflected in the surgical image; functioning as a display setting means for setting a display mode in which the drawn image is displayed on a display means in a predetermined mode; A program in which the inference model, the drawing manner, and the display manner are each set individually.
[0031] According to the invention (9), the same effects as those of the invention (1) are achieved. [Effects of the Invention]
[0032] According to the present invention, a surgical image processing platform and computer program can be provided that allow for the free setting of models to be implemented in layers that perform analysis, drawing, and display processes in the processing of surgical images. [Brief explanation of the drawings]
[0033] [Figure 1] 1 is a diagram showing the functional configuration of a surgery support system to which a surgery image processing platform according to an embodiment of the present invention is applied. [Figure 2] 1 is an example of a display mode by a surgical image processing platform according to an embodiment of the present invention. [Figure 3] FIG. 1 is a diagram illustrating an overview of a surgical image processing platform according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram illustrating data specifications in a surgical image processing platform according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating the flow of data in a surgical image processing platform according to an embodiment of the present invention. [Figure 6] FIG. 1 is a diagram illustrating the configuration of an inference platform of a surgical image processing platform according to an embodiment of the present invention. [Figure 7] FIG. 1 is a diagram illustrating the configuration of an inference platform of a surgical image processing platform according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram illustrating the configuration of a computing platform of a surgical image processing platform according to an embodiment of the present invention. [Figure 9] FIG. 1 is a diagram illustrating the configuration of a computing platform of a surgical image processing platform according to an embodiment of the present invention. [Figure 10] FIG. 1 is a diagram illustrating the settings in the display platform of the surgical image processing platform according to an embodiment of the present invention. [Figure 11] FIG. 1 is a diagram illustrating the flow of data in a surgical image processing platform according to an application example of an embodiment of the present invention. [Figure 12] FIG. 10 is a diagram illustrating the settings in the pre-processing platform of the surgical image processing platform according to an application example of an embodiment of the present invention. [Figure 13] FIG. 10 is a diagram illustrating the settings in the inference platform of the surgical image processing platform according to an application example of an embodiment of the present invention. [Figure 14] A figure explaining the inference model equipment information output by the inference platform to the preprocessing platform in a surgical image processing platform relating to an application example of an embodiment of the present invention. [Figure 15] 10 is a diagram illustrating connected device information output by the display platform to the pre-processing platform in a surgical image processing platform according to an application example of an embodiment of the present invention. FIG. DETAILED DESCRIPTION OF THE INVENTION
[0034] Hereinafter, embodiments of the present invention (hereinafter referred to as "embodiments") will be described in detail with reference to the accompanying drawings. In the following drawings, the same elements are designated by the same numbers or symbols throughout the description of the embodiments.
[0035] (Surgery support system) FIG. 1 is a diagram showing the functional configuration of a surgery support system to which a surgery image processing platform according to an embodiment of the present invention is applied. The surgical image processing platform 1 (inference platform 10, calculation platform 20, display platform 30) according to an embodiment of the present invention is applied to a surgical support system that supports surgeons at medical institutions (e.g., institutions where surgeons perform surgery, such as hospitals) by processing and displaying surgical images, which are images captured of the state of surgery performed by surgeons.
[0036] Specifically, the surgery support system 100 acquires, in an acquisition unit (such as a Camera Capture Module) that acquires surgical images, surgical images that are images of a surgeon performing surgery at a medical institution from an external device (such as a Camera & Imager) via a connection unit (such as a Camera Driver integrated IF). The surgical images capture images of the patient's body during surgery and instruments (for example, forceps, electric scissors, electric scalpels, energy devices such as ultrasonic coagulation and cutting devices, etc.) operated by the surgeon or assistant.
[0037] In addition to the acquisition unit and the surgical image processing platform 1 that processes surgical images, the surgical support system 100 also includes a system control unit that controls the entire surgical support system 100 (Ubuntu OS / Kernel, various system libraries, various connected devices (various sensors, various devices), etc.), various modules that process data from these devices, and connection units with various connected devices (camera driver integrated IF, sensor integrated IF, etc.).
[0038] The functional configuration of the surgery assistance system 100 is merely an example. A single functional block (database and functional processing unit) may be divided, or multiple functional blocks may be combined into a single functional block. Each functional processing unit is implemented by a central processing unit (CPU) (as a first control unit) or a graphics processing unit (GPU) (as a second control unit) built into the device or terminal. The central processing unit (CPU) or the graphics processing unit (GPU) reads a computer program (e.g., a core software or an application that causes the CPU to execute the various processes described above) stored in a storage device (storage unit) such as a read-only memory (ROM), flash memory, solid-state drive (SSD), or hard disk, and executes the computer program. Each functional processing unit may be implemented using a field-programmable gate array (FPGA). That is, each functional processing unit is implemented by the computer program reading and writing necessary data, such as tables, from a database (DB) stored in the storage device or a storage area in memory, and, in some cases, controlling related hardware (input / output devices, display devices, communication interface devices). Furthermore, the database (DB) in the embodiment of the present invention may be a commercial database, but may also refer to a simple collection of tables and files, and the internal structure of the database itself is not important.
[0039] In addition, in the surgical support system 100, the CPU as the first control unit mainly realizes the functions of the system control unit, and the GPU as the second control unit mainly realizes the functions of the surgical image processing platform 1 (inference platform 10, calculation platform 20, display platform 30).
[0040] (Surgical image processing platform) The surgical image processing platform 1 (inference platform 10, calculation platform 20, display platform 30) processes surgical images acquired by an acquisition unit (e.g., camera capture module). The surgical image processing platform 1 also classifies data for each external device (e.g., camera & imager) and records the data in a database according to an internal clock. This allows the user to check data for a single item or multiple items in chronological order.
[0041] The inference platform 10, which is an example of an inference means, inputs physical information indicating the state of the body and / or instrument information indicating the state of the instrument operated by the surgeon in the surgical image into an inference model, analyzes it using AI (Artificial Intelligence), and infers the anatomical structure, the structure of the anatomical structure, the trajectory of the instrument, the state of the instrument relative to the anatomical structure, etc. The inference model of the inference platform 10 is set by the user of the surgery support system 100.
[0042] The computation platform 20, an example of a computing means, generates a rendered image in which the analysis results of the inference platform 10 (such as anatomical structures, the structure of the anatomical structures, the trajectories of the instruments, and the state of the instruments relative to the anatomical structures) are rendered in a predetermined rendering style and reflected in the surgical image. In this embodiment, the rendering style is a style in which the analysis results of the inference platform 10 are displayed superimposed on the surgical image (for example, a style in which a part analyzed as a specific organ or body fluid is colored differently depending on the type of organ or body fluid). The rendering style also includes information indicating other aspects of assisting the surgeon in the surgery (for example, information indicating the elapsed time of the surgery, the ideal trajectory of the instruments, and information indicating the timing of resection of the resected organ). Such rendering styles are realized by a computation processing unit, which is an algorithm or the like that generates such styles. That is, a user of the surgery support system 100 sets the rendering style by selecting a computation processing unit.
[0043] The display platform (also referred to as a "UI platform") 30, which is an example of a display setting means, displays the drawing image generated by the computation platform 20 on a display means (console / LCD (Liquid Crystal Display) or the like) in a predetermined display mode. In this embodiment, the display mode is the way in which the drawing image is displayed (for example, displaying the original surgical image and the drawing image generated by the computation platform 20 side by side). Such a display mode is realized by a display processing unit, which is a UI (User Interface) or the like, which is an operation screen for the user. In other words, a user using the surgery assistance system 100 sets the display mode by selecting a display processing unit.
[0044] FIG. 2 is an example of a display mode by the surgical image processing platform according to the embodiment of the present invention. In the surgical image processing platform 1, the inference platform 10 analyzes the surgical image acquired by the acquisition unit (such as a camera capture module) (in the example shown in FIG. 2, it analyzes the parts that will become nervous tissue). Next, the calculation platform 20 generates a drawn image that reflects the analysis results in a predetermined drawing mode (in the example shown in FIG. 2, it colors the parts analyzed as nervous tissue) and reflects the drawn image in the surgical image. Then, the display platform 30 displays the drawn image on a display means (such as a console / LCD (Liquid Crystal Display)) in a predetermined display mode (in the example shown in FIG. 2, it displays the original surgical image and the drawn image generated by the calculation platform 20 side by side).
[0045] FIG. 3 is a diagram illustrating an overview of a surgical image processing platform according to an embodiment of the present invention. The surgical image processing platform 1 (inference platform 10, calculation platform 20, display platform 30) has an inference model, a rendering mode (calculation processing unit), and a display mode (display processing unit) that are individually set by the user. In the example shown in FIG. 3, the diagram on the left shows a setting example for user company X, and the diagram on the right shows a setting example for user company Y.
[0046] The inference platform 10 is configured by the user, Company X, with two inference models: Model A, the company's own model, and Model F, a model exclusive to Company X. Furthermore, the calculation platform 20 is set by Company X as a drawing mode (calculation processing unit) with Algo A, which is the company's own drawing algorithm, and Algo D, which is an open drawing algorithm of another company. Furthermore, the display platform 30 is set by company X to a GUID of another company's open GUI (Graphical User Interface) as a display mode (display processing unit).
[0047] Then, in the inference platform 10, Company X configures model A to receive data (such as surgical images) from endoscope A, and model F to receive data (such as device location information) from sensor A. Furthermore, in the calculation platform 20, the data (analysis results) from the model A is set to be input to the Algo A, and the data (analysis results) from the model F is set to be input to the Algo D. Moreover, in the display platform 30, the GUID is set to receive data (drawing images) from Algo A and Algo D. Moreover, in the display platform 30, the data from the GUID is set to be output to a display means (LCD (Liquid Crystal Display)) and a memory means (storage).
[0048] On the other hand, the inference platform 10 is set by the user, Company Y, as inference models, which are the company's own model, Model B, and the open model ** Doctor Model and other company's model. Furthermore, Company Y has set Algorithm F, which is an algorithm exclusive to Company Y, as the rendering mode (calculation processing unit) in the calculation platform 20. Furthermore, the display platform 30 is set by Company Y to have a GUIF dedicated to Company Y as the display mode (display processing unit).
[0049] In the inference platform 10, Company Y configures Model B to receive data from Device A (such as device location information), and configures the doctor model and other company models to receive data from Endoscope A and Endoscope B (such as surgical images). Furthermore, the calculation platform 20 is set so that data (analysis results) from Model B, **Doctor Model, and other company's model are input to Algo F. The calculation platform 20 is also set so that data from Algo F is output (feedback) to Device A. Moreover, in the display platform 30, the GUIF is set to receive data (drawing image) from Argo F. Moreover, in the display platform 30, the data from the GUIF is set to be output to a display means (console) and a memory means (storage).
[0050] In this way, in the surgical image processing platform, multiple inference models can be set on the inference platform 10, multiple drawing modes (arithmetic processing units) can be set on the calculation platform 20, and multiple display modes (display processing units) can be set on the display platform 30.
[0051] Furthermore, in the inference platform 10, data from multiple external devices (e.g., an endoscopic camera, a sensor provided on a device, etc.) may be input to multiple inference models, or data from multiple external devices may be input to a single inference model.
[0052] Furthermore, in the computation platform 20, data (analysis results) from multiple inference models may be input to multiple rendering modes (computation processing units), respectively, or data from multiple inference models may be input to one rendering mode (computation processing unit). Furthermore, in the computation platform 20, data from the rendering modes (computation processing units) is mainly output to the display platform 30, but may also be output to an external device other than the display platform 30.
[0053] Furthermore, in the display platform 30, data (drawn images) from multiple drawing modes (arithmetic processing units) may be input to multiple display modes (display processing units), respectively, or data from multiple drawing modes (arithmetic processing units) may be input to one display mode (display processing unit). Furthermore, in the display platform 30, data from the display modes (display processing units) is mainly output to a display means (console / LCD (Liquid Crystal Display) etc.), but it may also be output to a storage device, the inference platform 10 or the arithmetic platform 20, without being limited to the display means.
[0054] FIG. 4 is a diagram illustrating the data specifications in the surgical image processing platform according to the embodiment of the present invention. The surgical image processing platform 1 specifies the input data format that can be input, the output data format that can be output, and the implementation files that can be configured (the inference model of the inference platform 10, the calculation processing unit of the calculation platform 20, and the display processing unit of the display platform 30) for each layer (the inference platform 10, the calculation platform 20, and the display processing unit of the display platform 30).
[0055] In the example shown in Figure 4, in the inference platform 10, the input data format is preprocessed input image data deployed on a GPU, and the size and dtype of the data are specified. Also, the output data format showing the analysis results is confidence data deployed on a GPU, and the size and dtype of the data are specified. Also, the implementation file is specified as a general-purpose format (a model converted to ONNX).
[0056] In the computation platform 20, the input data format is defined to be the same as the output data format of the inference platform 10. This makes it possible to input data output from the inference platform 10 to the computation platform 20. In addition, the output data format showing a drawing image is a display image developed on a GPU, and the size and dtype of the data are defined. In addition, the implementation file is defined as a computation file written in a specified language.
[0057] In the display platform 30, the input data format is defined to be the same as the output data format of the computation platform 20. This makes it possible to input data output from the computation platform 20 to the display platform 30. The output data format indicating the display mode is a data group including the display and event processing of the designated library. The implementation file is defined as a UI file already described in the designated library.
[0058] In this way, by specifying the input data format that can be input, the output data format that can be output, and the implementation files that can be set for each of the inference platform 10, the calculation platform 20, and the display platform 30, it becomes possible to set not only content (inference model, calculation processing unit, display processing unit) prepared in advance by the provider of the surgical support system 100, but also user-specific content, content provided by third parties, and content provided as open content on the surgical image processing platform 1, as long as the content conforms to these specifications.
[0059] FIG. 5 is a diagram illustrating the flow of data in the surgical image processing platform according to the embodiment of the present invention. The inference platform 10, the computation platform 20, and the display platform 30 each have an input port for inputting data and an output port for outputting data that are individually set by the user. Furthermore, the inference platform 10, the computation platform 20, and the display platform 30 can each be set by the user to have multiple input ports and multiple output ports.
[0060] The input ports (Port II1, Port II2, etc. shown in FIG. 5) set in the inference platform 10 are connected by the user to connection parts (Device FI1, Camera IF11, etc.) connected to external devices (Camera & Imager, etc.), and data (surgical images (physical information, instrument information)) from the external devices are input via the connection parts. This input data is supplied to the inference model.
[0061] The output ports (PortIO1, PortIO2, etc. shown in Figure 5) set on the inference platform 10 are connected by the user to input ports (PortDI1, PortDI2, etc. shown in Figure 5) of the calculation platform 20, input ports of the display platform 30, or IFs of storage means or external devices, and output data (analysis results) from the inference model to these.
[0062] The input ports (PortDI1, PortDI2, etc. shown in FIG. 5) set in the calculation platform 20 are connected by the user to the output ports (PortIO1, PortIO2, etc. shown in FIG. 5) of the inference platform 10 and the output ports (PortGO3, etc. shown in FIG. 5) of the display platform 30, and output data (analysis results) from the inference platform 10 and output data (display format) from the display platform 30 are input. This input data is supplied to the calculation processing unit. The output ports (PortDO1, PortDO2, etc. shown in Figure 5) set on the calculation platform 20 are connected by the user to the input ports (PortGI1 shown in Figure 5) of the display platform 30 or the IF of a storage means or an external device (DeviceIFO1 shown in Figure 5), and output data (drawn image) from the calculation processing unit to these.
[0063] An input port (PortGI1 shown in FIG. 5) set in the display platform 30 is connected by the user to an output port (PortDO1 shown in FIG. 5) of the computation platform 20, and output data (drawn image) is input from the computation platform 20. This input data is supplied to the display processing unit. The output ports set on the display platform 30 (PortGO1, PortGO2, PortGO3, etc. shown in Figure 5) are connected by the user to a display means (LCD / console), an input port of the inference platform 10 (PortII4 shown in Figure 5), an input port of the calculation platform 20 (PortDI4 shown in Figure 5), or an IF of a storage means or an external device, and output data (display mode) from the display processing unit to these.
[0064] Next, the setting of content (inference model, calculation processing unit, display processing unit, etc.) by the user in the surgical image processing platform 1 will be described. 6 and 7 are diagrams illustrating the settings in the inference platform of the surgical image processing platform according to the embodiment of the present invention.
[0065] When setting content such as an inference model, other computational elements, input ports, and output ports in the inference platform 10, the user performs a setting operation. The setting operation is performed, for example, by operating the add button (the part displayed as "+" in the example shown in FIG. 6) displayed on the setting screen displayed on the display means by the display platform 30 functioning as the setting means. This causes the inference platform 10 to set input ports (PortII1, etc. in the example shown in FIG. 6) and output ports (PortIO1, etc. in the example shown in FIG. 6) one by one. Furthermore, when the add button is operated by the user, the display platform 30 functioning as the setting means displays on the display means a setting selection screen from which multiple types of content can be selected.
[0066] The setting selection screen in the inference platform 10 displays the following content that can be selected by the user: multiple types of inference models (in the example shown in Figure 6, basic model, open model, closed model, etc.), multiple types of calculation methods for the analysis results of the inference models (in the example shown in Figure 6, ADD, MATMUL, ABS, etc.), multiple types of constants (in the example shown in Figure 6, SCALAR, etc.), and conditional branching (in the example shown in Figure 6, SELECTOR, etc.).
[0067] On the setting selection screen, the contents that can be selected by the user may be grouped by supplier, by analysis target (for example, anatomical structure, etc.), or by type of surgery. In the example shown in FIG. 6, on the setting selection screen, multiple inference models are grouped by source (basic model, open model, closed model). In the example shown in Fig. 7, multiple inference models are grouped into multiple layers on the setting selection screen. In detail, in the example shown in Fig. 7, in the top layer, they are grouped by analysis object (connective tissue, nerves, etc.), and within each group, they are further grouped by surgical content (thoracoscopic lung resection, gastric / colon region of robot-assisted endoscopic surgery, etc.).
[0068] For example, a user can set content (inference model, calculation method, conditional branching, etc.) in the inference platform 10 by selecting content on a setting selection screen and dragging the selected content into the inference platform 10 on the setting screen.
[0069] Next, the user performs an operation on the setting screen to connect the content, input ports, and output ports set in the inference platform 10 (for example, in the example shown in Figure 6, an operation such as dragging the source of the connection (e.g., Port II1) to the destination of the connection (e.g., Model B)), thereby connecting the source of the connection and the destination of the connection, and data flows from the source of the connection to the destination of the connection.
[0070] 8 and 9 are diagrams illustrating the settings in the computing platform of the surgical image processing platform according to the embodiment of the present invention.
[0071] A user performs a setting operation when setting content such as a processing unit, other processing elements, input ports, and output ports on the computing platform 20. The setting operation is performed, for example, by operating an add button (the portion displayed as "+" in the example shown in FIG. 8) displayed on a setting screen displayed on the display means by the display platform 30 functioning as setting means. This causes the computing platform 20 to set input ports (PortDI1, etc. in the example shown in FIG. 8) and output ports (PortDO1, etc. in the example shown in FIG. 8) one by one. Furthermore, when the add button is operated by the user, the display platform 30 functioning as setting means displays on the display means a setting selection screen from which multiple types of content can be selected.
[0072] The setting selection screen in the calculation platform 20 displays, as content that can be selected by the user, multiple types of calculation processing units (in the example shown in FIG. 8, basic model, open model, closed model, etc.), various processes for the processing results of the calculation processing units (for example, multiple types of filters (in the example shown in FIG. 8, low pass, high pass, band pass, etc.), multiple types of analysis methods (in the example shown in FIG. 8, peak, histogram, FFT, etc.), multiple types of calculation methods (in the example shown in FIG. 8, ADD, MATMUL, ABS, etc.), and constants (in the example shown in FIG. 8, SCALAR, etc.)).
[0073] On the setting selection screen, the content that can be selected by the user may be grouped by supplier, by analysis target (for example, anatomical structure, etc.), by surgical content, or by graphical representation. In the example shown in FIG. 8, on the setting selection screen, a plurality of processing units are grouped by supplier (basic model, open model, closed model).
[0074] Furthermore, the content of the drawing expression realized by the arithmetic processing unit may be adjustable on the setting selection screen. In the example shown in Figure 9, on the setting selection screen, multiple calculation processing units can be selected for each drawing expression (drawing color, confidence threshold, opacity, drawing method, blinking display, etc.), and for each drawing expression item, the drawing expression can be adjusted (for example, by entering a numerical value or selecting a color, etc.).
[0075] For example, a user can select content on a setting selection screen and drag the selected content into the computing platform 20 on the setting screen, thereby setting content (computing processing unit, computing method, conditional branching, etc.) on the computing platform 20.
[0076] Next, the user performs an operation on the setting screen to connect the content, input ports, and output ports set in the computing platform 20 (for example, in the example shown in Figure 8, an operation such as dragging the connection source (e.g., PortDI1 to PortDI4) to the connection destination (e.g., model F)), thereby connecting the connection source and the connection destination and allowing data to flow from the connection source to the connection destination.
[0077] FIG. 10 is a diagram illustrating the settings in the display platform of the surgical image processing platform according to an embodiment of the present invention.
[0078] A user performs a setting operation when setting a display processing unit, other calculation elements, or content such as input ports and output ports on the display platform 30. The setting operation is performed, for example, by operating an add button (the portion displayed as "+" in the example shown in FIG. 10) displayed on a setting screen displayed on the display means by the display platform 30 functioning as setting means. This causes the display platform 30 to set input ports (PortGI1, etc. in the example shown in FIG. 10) and output ports (PortGO1, etc. in the example shown in FIG. 10) one by one. Furthermore, when the add button is operated by the user, the display platform 30 functioning as setting means displays on the display means a setting selection screen from which multiple types of content can be selected.
[0079] The setting selection screen on the display platform 30 displays, as content that can be selected by the user, multiple types of display processing units (in the example shown in Figure 8, basic model, open model, closed model, etc.), multiple types of calculation methods for the processing results of the display processing units (in the example shown in Figure 8, ADD, MATMUL, ABS, etc.), and multiple types of conditional branches (in the example shown in Figure 8, SELECTOR, etc.).
[0080] On the setting selection screen, the contents that can be selected by the user may be grouped by supplier, by analysis target (for example, anatomical structure, etc.), by type of surgery, or by display mode. In the example shown in FIG. 10, on the setting selection screen, a plurality of display processing units are grouped by supplier (basic model, open model, closed model).
[0081] For example, a user can select content on a setting selection screen and drag the selected content into the display platform 30 on the setting screen, thereby setting content (display processing unit, calculation method, conditional branching, etc.) on the display platform 30.
[0082] Next, the user performs an operation on the setting screen to connect the content, input ports, and output ports set on the display platform 30 (for example, in the example shown in Figure 10, an operation such as dragging the source of the connection (e.g., PortGI1) to the destination of the connection (e.g., GUI F)), thereby connecting the source of the connection and the destination of the connection and allowing data to flow from the source of the connection to the destination of the connection.
[0083] Once the user has completed the settings for the inference platform 10, the computation platform 20, and the display platform 30, the user connects the platforms together (for example, connecting the output port of the inference platform 10 (for example, PortIO1) to the input port of the computation platform 20 (for example, PortDI1)) and connects the surgical image processing platform to external devices (for example, connecting DeviceIFI1 to the input port of the inference platform 10 (for example, PortII1)) as shown in Figure 5. By following the above procedure, each user can build their own unique surgical assistance system.
[0084] In addition to the above functions, the surgical image processing platform 1 may output the test results of the constructed platform as a template for submitting to a specified organization for standards application. Furthermore, when introducing an inference model, a calculation processing unit, or a display processing unit other than those provided in the surgical image processing platform 1, the surgical image processing platform 1 may be provided with a function for confirming the performance of these units.
[0085] According to such a surgical image processing platform 1, a user of the surgical image processing platform 1 can, for example, individually set their own inference model, drawing style, and display style, thereby analyzing surgical images in their own way, generating drawing images in their own way based on the analysis results, and displaying these drawing images in their own way. Therefore, it is possible to provide a surgical image processing platform that allows for the free setting of models to be implemented in layers that perform analysis, drawing, and display processing in the processing of surgical images.
[0086] Furthermore, according to the surgical image processing platform 1, it is possible to set multiple inference models in the inference platform 10, multiple rendering modes in the calculation platform 20, and multiple display modes in the display platform 30. This makes it possible to equip each layer (the inference platform 10, the calculation platform 20, and the display platform 30) in the processing of surgical images with a variety of functions (functions realized by each model).
[0087] Furthermore, according to the surgical image processing platform 1, the input port for inputting data and the output port for outputting data can be set individually in the inference platform 10, the calculation platform 20, and the display platform 30, so that the data input source and the data output destination can be set individually in each layer (the inference platform 10, the calculation platform 20, and the display platform 30) in the processing of surgical images. This increases the degree of freedom in the input source and output destination of each layer.
[0088] Furthermore, according to the surgical image processing platform 1, multiple input ports and multiple output ports can be set for each of the inference platform 10, the calculation platform 20, and the display platform 30. This allows various data to be input and data to be output in various directions (for example, to another device or another layer) at each layer in the surgical image processing (the inference platform 10, the calculation platform 20, and the display platform 30).
[0089] Furthermore, according to the surgical image processing platform 1, in addition to data from an external device, data from the display platform 30 can be input to the inference platform 10. Furthermore, in addition to data from the inference platform 10, data from the display platform 30 can be input to the calculation platform 20. This makes it possible to feed back data from a platform located downstream in the processing (for example, the inference platform 10 or the display platform 30 for the calculation platform 20) to a means located upstream in the processing (for example, the inference platform 10 or the calculation platform 20 for the display platform 30). This makes it possible to obtain multiple types of results, for example, by repeatedly using data input from an external device and processing it using different inference models or rendering modes.
[0090] [Application example] Next, application examples of the embodiment of the present invention will be described. In the following description, the same components as those in this embodiment will be denoted by the same reference numerals, and the description thereof will be omitted or simplified. FIG. 11 is a diagram illustrating the flow of data in a surgical image processing platform according to an application example of an embodiment of the present invention.
[0091] The surgical image processing platform 1A according to the application example includes a pre-processing platform 40, which is an example of a pre-processing means, in addition to the configuration of this embodiment (the inference platform 10, the calculation platform 20, and the display platform 30).
[0092] First, as described above, the inference platform 10 inputs physical information indicating the state of the body and / or instrument information indicating the state of the instruments operated by the surgeon from surgical images acquired by external devices (e.g., endoscopic systems / endoscopes, etc.) used in medical institutions into an inference model, analyzes the information using AI, and infers the anatomical structure, the structure of the anatomical structure, the trajectory of the instrument, the state of the instrument relative to the anatomical structure, etc.
[0093] Here, external devices (e.g., endoscope systems, endoscopes, etc.) used in medical institutions are provided by various manufacturers, and the types (models) used vary depending on the medical institution. Furthermore, different types of external devices result in different image quality of the acquired surgical images.
[0094] Furthermore, the inference model used by the inference platform 10 for analysis is trained using surgical images captured by a specific external device. For example, the external device that captured the surgical images on which the inference model learned may be different from the external device used by the medical institution that uses the surgical image processing platform 1 (the external device that provides the surgical images to the surgical image processing platform 1). In such cases, the image quality of the surgical images captured by these external devices also differs. If there is a large difference between the image quality of the surgical images on which the inference model learned and the image quality of the surgical images acquired when using the surgical image processing platform 1, the accuracy of the analysis results of the inference platform 10 may decrease.
[0095] The preprocessing platform 40 reduces the discrepancy between the image quality of the surgical image input to the surgical image processing platform 1 and the image quality of the surgical image learned by the inference model, thereby preventing a decrease in the accuracy of the analysis results of the inference platform 10.
[0096] The preprocessing platform 40, an example of a preprocessing means, converts the image quality of a surgical image using a conversion formula corresponding to the image quality of the image learned by the inference model of the inference platform 10. Specifically, the preprocessing platform 40 has a camera image quality converter, which converts the image quality of the surgical image acquired by an acquisition unit (such as a Camera Capture Module (see FIG. 1)) into an image quality similar to that of the surgical image learned by the inference model of the inference platform 10, depending on the external device that captured the surgical image, generates a preprocessed surgical image with converted image quality, and provides it to the inference platform 10. The conversion formula for the preprocessing platform 40 is set by a user of the surgery support system 100 or automatically, as described below. Note that the "conversion formula" is not limited to one that converts the image quality of the surgical image into an image quality similar to that of the surgical image learned by the inference model of the inference platform 10. The "conversion formula" can be any method that converts the image quality of the surgical image into an image quality that improves the analysis accuracy of the inference platform 10. For example, it may be a method that performs frequency analysis on the image and converts the image using a frequency approach.
[0097] Furthermore, in the pre-processing platform 40, similar to the inference platform 10, the computation platform 20, and the display platform 30, the input ports (PortPPI1, PortPPI2, etc. shown in FIG. 11) through which data is input and the output ports (PortPPO1, PortPPO2 shown in FIG. 11) through which data is output are set individually by the user. Also, similar to the inference platform 10, the computation platform 20, and the display platform 30, the pre-processing platform 40 allows the user to set multiple input ports and multiple output ports.
[0098] The flow of data in the surgical image processing platform 1A will be described with reference to FIG. In the surgical image processing platform 1A, input ports (PortPPI1, PortPPI2, etc. shown in FIG. 11) set in the pre-processing platform 40 are connected by the user to connectors (Device FI1, Camera IF11, etc.) connected to external devices (Camera & Imager, etc.), and data (surgical images (physical information, instrument information)) from the external devices (e.g., endoscope systems / endoscopes, etc.) are input via the connectors. In addition, input ports (PortPPI4, PortPPI5, etc. shown in FIG. 11) set in the pre-processing platform 40 are connected to output ports (PortIO3 shown in FIG. 11) of the inference platform 10 and output ports (PortGO4 shown in FIG. 11) of the display platform 30, and output data (analysis results) from the inference platform 10 and output data (display format) from the display platform 30 are input. This input data is supplied to a camera image quality converter and converted to image quality approximating the image quality of the surgical images learned by the inference model of the inference platform 10.
[0099] The output ports (PortPPO1 and PortPPO2 shown in Figure 11) set in the preprocessing platform 40 are connected by the user to the input ports (PortII1 and PortII2 shown in Figure 11) of the inference platform 10, and output data (preprocessed surgical images) to the inference platform 10.
[0100] The input ports (Port II1, Port II2, etc. shown in FIG. 11) set in the inference platform 10 are connected by the user to the output ports (Port PPO1, Port PPO2 shown in FIG. 11) of the preprocessing platform 40, and output data (preprocessed surgical images) are input. This input data is supplied to the inference model.
[0101] The output ports (PortIO1, PortIO2, etc. shown in Figure 11) set on the inference platform 10 are connected by the user to input ports (PortDI1, PortDI2, etc. shown in Figure 11) on the computation platform 20 or input ports (PortPPI4, etc. shown in Figure 11) on the preprocessing platform 40, and output data (analysis results) from the inference model are output to these.
[0102] The input ports (PortDI1, PortDI2, etc. shown in FIG. 11) set in the calculation platform 20 are connected by the user to the output ports (PortIO1, PortIO2, etc. shown in FIG. 11) of the inference platform 10 and the output port (PortGO3 shown in FIG. 11) of the display platform 30, and output data (analysis results) from the inference platform 10 and output data (display format) from the display platform 30 are input. This input data is supplied to the calculation processing unit.
[0103] The output port (PortDO1 shown in Figure 11) set on the computation platform 20 is connected by the user to the input port (PortGI1 shown in Figure 11) of the display platform 30, and output data (drawn image) from the computation processing unit is output to these.
[0104] An input port (PortGI1 shown in FIG. 11) set in the display platform 30 is connected by the user to an output port (PortDO1 shown in FIG. 11) of the computation platform 20, and output data (drawn image) is input from the computation platform 20. This input data is supplied to the display processing unit.
[0105] The output ports set in the display platform 30 (PortGO1, PortGO2, PortGO3, etc. shown in Figure 11) are connected by the user to a display means (LCD / console), an input port of the inference platform 10 (PortII3 shown in Figure 11), an input port of the calculation platform 20 (PortDI3 shown in Figure 11), or an input port of the pre-processing platform 40 (PortPPI5, etc. shown in Figure 11), and output data (display format) from the display processing unit is output to these.
[0106] Next, the setting of content (camera image quality converter, inference model, etc.) by the user in the surgical image processing platform 1A will be described. FIG. 12 is a diagram illustrating the settings in the pre-processing platform of the surgical image processing platform according to an application example of the embodiment of the present invention.
[0107] The user performs a setting operation when setting content such as a camera image quality converter, other calculation elements, input ports, and output ports in the pre-processing platform 40. The setting operation is performed, for example, by operating an add button (the portion displayed as "+" in the example shown in FIG. 12) displayed on a setting screen displayed on the display means by the display platform 30 functioning as setting means. This causes the pre-processing platform 40 to set input ports (PortPPI1, etc. in the example shown in FIG. 12) and output ports (PortPPO1, etc. in the example shown in FIG. 12) one by one. Furthermore, when the add button is operated by the user, the display platform 30 functioning as setting means displays on the display means a setting selection screen from which multiple types of content can be selected.
[0108] The setting selection screen in the preprocessing platform 40 displays the following content that can be selected by the user: multiple types of conversion elements included in the camera image quality converter, multiple types of preprocessing for the conversion results of the camera image quality converter (in the example shown in Figure 12, Normalize, Standardize, Grayscale, Binalize, etc.), and conditional branching (in the example shown in Figure 12, SELECTOR, etc.).
[0109] The camera image quality converter is a database that stores a series of conversion formulas for converting the source image into image quality (spatial frequency, brightness, color tone, etc.) equivalent to the destination image. The pre-processing platform 40 may acquire (read) the camera image quality converter from a storage means of the surgery support system 100 (e.g., the database shown in FIG. 1 ) or may acquire (download) it from an external server (e.g., the provider of the surgical image processing platform 1A or another user of the surgical image processing platform 1A). The pre-processing platform 40 may output (store) the camera image quality converter to a storage means of the surgery support system 100 (e.g., the database shown in FIG. 1 ) or may be capable of outputting (transmitting) it to an external computer (e.g., the provider of the surgical image processing platform 1A or another user of the surgical image processing platform 1A). In detail, the camera image quality converter includes, as conversion elements, an "endoscopic system (S) + endoscope (ES) conversion element" and an "endoscopic system (S) setting value conversion element."
[0110] In the "endoscopic system (S) + endoscope (ES) conversion element," multiple types of conversion destinations (combinations of an endoscopic system and an endoscope that are connected to the surgical image processing platform 1A and acquire surgical images) are associated with multiple types of conversion sources (combinations of an endoscopic system and an endoscope that captured the surgical images learned by the inference model), respectively. In the example shown in Figure 12, "Conversion A" is associated with the combination of the destination endoscopic system (S:A) and endoscope (ES:A) as a conversion formula that approximates the image quality (spatial frequency, brightness, color tone, etc.) of the surgical image obtained by this combination to the image quality of the surgical image obtained by the source combination of the endoscopic system (S:A) and endoscope (ES:B).
[0111] In the "endoscopic system (S) setting value conversion element," multiple types of conversion sources (setting values used in the endoscopic system connected to the surgical image processing platform 1A and acquiring surgical images) are associated with multiple types of conversion destinations (default setting values of the endoscopic system connected to the surgical image processing platform 1A and acquiring surgical images). In the example shown in Figure 12, information indicating values for each of multiple types of conversion destinations ("brightness," "hue," "color mode," and "contrast") is associated with each of multiple types of setting values.
[0112] For example, the user can select content on a setting selection screen and drag the selected content into the pre-processing platform 40 on the setting screen, thereby setting the content (conversion formula, calculation method, conditional branching, etc.) in the pre-processing platform 40. Note that, as will be described in detail later, the pre-processing platform 40 may automatically set the conversion formula.
[0113] Next, the user performs an operation on the setting screen to connect the content, input ports, and output ports set in the pre-processing platform 40 (for example, in the example shown in Figure 12, an operation such as dragging multiple connection sources (e.g., PortPPI1 to PortPPI5) to multiple connection destinations (e.g., "Conversion C" and "Conversion A")), thereby connecting the connection sources and destinations and allowing data to flow from the connection sources to the connection destinations.
[0114] FIG. 13 is a diagram illustrating the settings in the inference platform of the surgical image processing platform according to an application example of the embodiment of the present invention. In the setting selection screen of the inference platform 10 according to the application example, in addition to the example shown in Fig. 6, each inference model is grouped by the combination of the type of endoscopic system (S) and the type of endoscope (ES) that captured the surgical images from which it learned. In the example shown in Fig. 13, for example, Model A, which is a basic model, Model **, which is an open model, and Model F, which is a closed model, are grouped into information (S:A+ES:A) that indicates the type of endoscopic system and the type of endoscope that captured the surgical images from which these models (inference models) learned. In other words, the inference platform 10 stores each inference model in association with the combination of the type of endoscopic system (S) and the type of endoscope (ES) that captured the surgical images from which each inference model learned.
[0115] The display platform 30 in the application example may be connected to the surgical image processing platform 1A and may be capable of setting the type of endoscopic system (S) that captures surgical images, information indicating the type of endoscope (ES), and setting values of the endoscopic system (S) as setting items for each model (GUI A to F) in the display processing unit of the setting selection screen (see Figure 10).
[0116] In this case, the pre-processing platform 40 may acquire information indicating the type of endoscope system (S) used in the usage environment of the surgical image processing platform 1A, information indicating the type of endoscope (ES), and setting values of the endoscope system (S), which are inputted from an input port (Port PP5 in the example shown in FIG. 11) and set by the display platform 30. Note that the pre-processing platform 40 may acquire, as conversion destination information, information indicating the combination of the type of endoscope system (S) and the type of endoscope (ES) from the serial signal of the endoscope system inputted from an input port (Port PP1 in the example shown in FIG. 11). Also, the pre-processing platform 40 may acquire, as conversion source information, the setting values of the endoscope system (S) inputted from an input port (Port PP1 in the example shown in FIG. 11).
[0117] Next, an example in which the pre-processing platform 40 automatically sets the conversion formula will be described. The preprocessing platform 40 acquires, as source information for conversion, information indicating the combination of the type of endoscopic system (S) and endoscope (ES) learned by the inference model set in the inference platform 10, which is input from an input port (Port PP4 in the example shown in Figure 11).
[0118] FIG. 14 is a diagram illustrating inference model equipment information that the inference platform outputs to the pre-processing platform in a surgical image processing platform according to an application example of an embodiment of the present invention. The inference platform 10 outputs to the preprocessing platform 40 inference model equipment information indicating the combination of the type of endoscopic system (S) and the type of endoscope (ES) that captured the surgical image learned by the inference model selected by the user.
[0119] In the example shown in Figure 14, for example, if the inference model selected by the user in the inference platform 10 is model B, the inference platform 10 outputs information (S:A+ES:B) indicating the type of endoscopic system and the type of endoscope that captured the surgical image learned by model B to the preprocessing platform 40 as inference model equipment information.
[0120] FIG. 15 is a diagram illustrating connected device information that the display platform outputs to the pre-processing platform in the surgery image processing platform according to an application example of the embodiment of the present invention. The display platform 30 outputs to the pre-processing platform 40 information indicating the type of endoscopic system (S) that captures surgical images connected to the surgical image processing platform 1A, the type of endoscope (ES), and connected device information indicating the setting values of the endoscopic system (S) ("brightness," "color tone," "color mode," and "contrast").
[0121] In the example shown in Figure 15, for example, the display platform 30 outputs to the pre-processing platform 40 information (S:A) indicating the type of endoscopic system (S) that captures the surgical image, information (ES:A) indicating the type of endoscope (ES), and connected device information indicating the setting values of the endoscopic system (S) ("brightness" (Def (default)), "color tone" (R- / B+), "color mode" (Def), and "contrast" (high)).
[0122] In the above case, the pre-processing platform 40 acquires information (S:A+ES:B) indicating the type of endoscopic system (S) and the type of endoscope (ES) that captured the surgical images learned by model B in the inference model device information as information indicating the source of conversion. In addition, the pre-processing platform 40 acquires information (S:A+ES:A) indicating the type of endoscopic system (S) and the type of endoscope (ES) that captures the surgical images connected to the surgical image processing platform 1A in the connected device information of the example shown in Figure 15 as information indicating the destination of conversion.
[0123] In this case, in the example shown in Figure 12, the pre-processing platform 40 selects the conversion formula "Conversion A" as the "endoscopic system (S) + endoscope (ES) conversion element" in which the source (S:A + ES:B) and destination (S:A + ES:A) are associated.
[0124] In addition, in the example shown in Figure 12, the pre-processing platform 40 selects a conversion formula as the "endoscopic system (S) setting value conversion element" to convert the source ("brightness" (Def (default)), "color tone" (R- / B+), "color mode" (Def), and "contrast" (high)) into the destination (the default setting values of the endoscopic system (S:A) connected to the surgical image processing platform 1A and acquiring surgical images).
[0125] According to the surgical image processing platform 1A of this application example, the pre-processing platform 40 converts the image quality of the surgical image captured during surgery, and the inference platform 10 analyzes the surgical image with the converted image quality. As a result, rather than analyzing the surgical image input to the surgical image processing platform 1A as is, the image quality of the surgical image is converted to, for example, an image quality that improves the analytical accuracy of the inference platform 10, and the surgical image with the converted image quality is analyzed, thereby preventing a decrease in the accuracy of the analysis results.
[0126] Furthermore, the surgical image processing platform 1A makes it possible to provide a converter used by a user to, for example, the provider of the surgical image processing platform 1A or another user. This makes it possible to improve or reuse the converter that converts the image quality of the surgical image, thereby improving usability.
[0127] Furthermore, according to the surgical image processing platform 1A, the preprocessing platform 40 converts the image quality of the surgical image taken of the surgery to be analyzed using a conversion formula corresponding to the image quality learned by the inference platform 10, thereby preventing a decrease in the accuracy of the analysis results.
[0128] Although the present invention has been described above using embodiments, it goes without saying that the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. Furthermore, it is clear from the claims that such modifications and improvements are also included within the technical scope of the present invention. In the above embodiments, the present invention has been described as an invention of a surgical image processing platform, but it can also be considered as an invention of a program that causes the surgical image processing platform to function as various parts. [Explanation of symbols]
[0129] 1. 1A Surgical Imaging Platform 10 Inference Platform 20 Computing Platform 30 Display Platform 40 Pre-processing Platform 100 Surgical Support System
Claims
1. A surgical image processing platform for processing surgical images captured during a surgical procedure performed by a surgeon, comprising: an inference means for setting an inference model for analyzing the surgical image; A calculation means for setting a drawing mode for generating a drawing image in which the analysis result by the inference model is reflected in the surgical image; a display setting means for setting a display mode in which the drawn image is displayed on a display means in a predetermined mode, A surgical image processing platform characterized in that the inference model, the drawing manner, and the display manner are each set individually.
2. The inference means is capable of setting a plurality of the inference models, A plurality of drawing modes can be set in the calculation means, 2. The surgical image processing platform according to claim 1, wherein the display setting means is capable of setting a plurality of the display modes.
3. The surgical image processing platform according to claim 1, characterized in that the inference means, the calculation means and the display setting means each have an input port for inputting data and an output port for outputting data that are individually set.
4. 4. The surgical image processing platform according to claim 3, wherein a plurality of said input ports and a plurality of said output ports can be set in said inference means, said calculation means and said display setting means, respectively.
5. The input port of the inference means is An external device can be connected to the output port of the display setting means, The data output from the external device and the data output from the display setting means can be input, The input port of the computing means is a device that is connectable to the output port of the inference means and / or the display setting means; data output from the inference means and / or the display setting means; The input port of the display setting means is a signal output port of the computing means; 5. The surgical image processing platform according to claim 3, wherein data output from said calculation means can be input.
6. A pre-processing means for converting the image quality of the surgical image is further provided, The surgical image processing platform according to claim 1 , wherein the inference means is configured to set an inference model for analyzing the surgical image whose image quality has been converted.
7. The surgical image processing platform of claim 6, wherein the pre-processing means is capable of outputting a converter that converts the image quality of the surgical image.
8. The surgical image processing platform described in claim 6 or 7, characterized in that the pre-processing means converts the image quality of the surgical image using a conversion formula corresponding to the image quality of the image learned by the inference model.
9. A surgical image processing platform that processes surgical images taken during surgery by surgeons. an inference means for setting an inference model for analyzing the surgical image; A calculation means for setting a drawing mode for generating a drawing image in which the analysis result by the inference model is reflected in the surgical image; a display setting unit that sets a display mode in which the drawn image is displayed on a display unit in a predetermined mode; A program in which the inference model, the drawing manner, and the display manner are each set individually.
Citation Information
Patent Citations
Surgery support system, surgery support method, information processing device, and information processing program
JP2021029258A
