Information processing apparatus, assistance system

Abstract

The present application relates to an information processing apparatus, an auxiliary system having a first system and a second system separate from the first system, the second system having the information processing apparatus, in which an auxiliary system is provided with: a data transmission section that transmits drawing information based on an operation input, which is necessary to generate annotation image data, to the first system; an image generation section that generates annotation image data in accordance with the drawing information; and an image synthesis section that synthesizes the annotation image data generated by the image generation section with image data received from the first system.

Description

Technical Field

[0001] This invention relates to a technique for an instructor to guide a practitioner remotely. Background Technology

[0002] A known assistive system allows a surgeon to perform surgery on a patient while receiving instructions from a remote supervisor. This system is used, for example, when a less experienced surgeon is performing surgery under the guidance of an experienced supervising surgeon.

[0003] According to the invention disclosed in Patent Document 1, a monitor displaying the endoscopic image (hereinafter also referred to as the endoscopic image) of the endoscope inserted into the patient is set up remotely, allowing a remote supervising physician to input instructions while observing the endoscopic image displayed on the monitor. Furthermore, a monitor in the operating room displays a composite image formed by overlaying the remotely input instructions onto the endoscopic image, allowing the surgeon to perform the operation while receiving instructions from the remote supervising physician.

[0004] According to the aforementioned assistive system, even less experienced surgeons can reliably perform surgery on patients by having an instructor observe the camera images displayed on the monitor and receive instructions, thereby improving the surgeon's own skills.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2000-271147 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] In the case of using such an auxiliary system in fields such as endoscopic surgery where the safety of the surgical subject (patient) is required, it is important that the endoscopic image reflecting the instructions of the surgeon be displayed on the surgeon's monitor with the lowest possible latency so that the surgeon can monitor the patient's condition in real time.

[0010] In addition, in order for the instructor to input instructions without discomfort, it is desirable to display the endoscopic image reflecting the input on their own monitor with the lowest possible latency.

[0011] The present invention was made in view of the above circumstances, and its purpose is to provide an auxiliary system in which a remote instructor can provide appropriate assistance to the practitioner.

[0012] Solution for solving the problem

[0013] The present invention relates to an information processing apparatus comprising an auxiliary system having a first system and a second system separate from the first system, wherein the second system in the auxiliary system has the information processing apparatus, the information processing apparatus comprising: a data transmission unit that transmits drawing information based on operation input required to generate annotated image data to the first system; an image generation unit that generates annotated image data based on the drawing information; and an image synthesis unit that synthesizes the annotated image data generated by the image generation unit with image data received from the first system.

[0014] The annotated image data here is image data to be overlaid on endoscopic images, and is image data generated based on rendering information.

[0015] In addition, drawing information refers to information such as coordinates, line type, color, and line width specified through operation input.

[0016] In the first system of the aforementioned auxiliary system, annotation image data is generated based on the drawing information, and the generated annotation image data is combined with camera image data to generate composite image data. In the information processing device of the second system, the image synthesis unit combines the composite image data received from the first system with the annotation image data generated by the image generation unit.

[0017] Therefore, annotated image data are generated in the first system and the second system respectively.

[0018] Furthermore, it can be considered that in the first system, annotation image data is generated based on the drawing information, and multiple image data including the generated annotation image data are synthesized to generate composite image data. The image synthesis unit of the information processing device of the second system synthesizes the composite image data received from the first system with the annotation image data generated by the image generation unit.

[0019] Therefore, it is possible to confirm on the second system side whether the synthetic image data received from the first system reflects the drawing information sent to the first system side.

[0020] Furthermore, it can be considered that the image generation unit in the information processing device of the second system described above generates annotation image data that can be distinguished from the annotation image data contained in the synthetic image data.

[0021] Therefore, it is possible to distinguish between display based on annotated image data contained in the synthetic image data and display based on annotated image data generated by the image generation unit, thereby confirming whether annotated image data reflecting drawing information based on input in the second system has been generated on the first system side.

[0022] The present invention relates to an information processing apparatus comprising an auxiliary system having a first system and a second system separate from the first system, wherein the first system in the auxiliary system has the information processing apparatus, the information processing apparatus comprising: an image generation unit that receives drawing information based on operation input from the second system and generates annotated image data based on the received drawing information; an image compositing unit that combines the input image data with the annotated image data generated by the image generation unit to generate composite image data; and a data transmission unit that transmits the composite image data generated by the image compositing unit to the second system.

[0023] Here, whenever synthetic image data is generated on the first system side, drawing information with a smaller communication capacity compared to the annotated image data is received from the second system instead of receiving the annotated image data.

[0024] Furthermore, it can be considered that in the above-mentioned information processing device, the delay time of the image synthesis unit from the input of multiple image data to the output of the synthesized image data is set to be less than 30m seconds.

[0025] Therefore, when the input image data is displayed on the monitor on the first system side, the operator visually confirming the monitor will not feel any delay.

[0026] The auxiliary system of the present invention includes the information processing device of the first system and the information processing device of the second system described above. The information processing devices communicate with each other to realize the auxiliary system.

[0027] The effects of the invention

[0028] According to the present invention, instructions based on input from a remote instructor are displayed with very low latency on monitors on both the practitioner's and instructor's sides, thereby enabling the practitioner and instructor to share the instructions in real time. Attached Figure Description

[0029] Figure 1 This is a diagram illustrating an outline of the auxiliary system in an embodiment of the present invention.

[0030] Figure 2 This is a diagram showing an endoscopic image captured in this embodiment.

[0031] Figure 3This is a diagram showing a composite image created by overlaying an annotation image onto an endoscopic camera image in this embodiment.

[0032] Figure 4 This is a diagram that schematically illustrates an example of the structure of the auxiliary system in this embodiment.

[0033] Figure 5 This is a diagram showing the annotated images in this embodiment.

[0034] Figure 6 This is a diagram showing a composite image created by overlaying an annotation image onto an endoscopic camera image in this embodiment.

[0035] Figure 7 This is a diagram showing a composite image created by overlaying an annotation image onto an endoscopic camera image in this embodiment.

[0036] Figure 8 This is a diagram showing the annotated images in this embodiment.

[0037] Figure 9 This is a diagram showing a composite image created by overlaying an annotation image onto an endoscopic camera image in this embodiment. Detailed Implementation

[0038] The following is for reference Figures 1 to 9 To illustrate the implementation method.

[0039] Furthermore, the structures shown in the accompanying drawings are merely examples illustrating how to implement the present invention. Therefore, various modifications can be made based on design and other factors without departing from the technical concept of the present invention. Additionally, for structures already described, to avoid repetition, the same reference numerals may be added later and further description may be omitted.

[0040] <1. Overview of Auxiliary Systems>

[0041] Figure 1 The auxiliary system 100 is shown.

[0042] In this embodiment, as an example of the auxiliary system 100, a surgical assistance system is described in which the surgeon 1 in the operating room Rm1 can perform surgery on the patient 3 while confirming the instructions given by the supervising physician 2 in the guidance room Rm2, which is separate from the operating room Rm1. In this surgical assistance system, for example, the surgeon 1 is the surgeon performing the operation on the patient 3, and the supervising physician 2 is the supervising physician who guides the surgeon performing the operation.

[0043] For example, an endoscope 11 is inserted into the body cavity of patient 3, thereby... Figure 2The endoscopic image 30 inside the body cavity, as shown, is displayed on the monitor 14. The operator 1 can confirm the endoscopic image 30 on the monitor 14.

[0044] The endoscopic image 30 is also displayed on the monitor 25 of the guidance terminal 26 located in the guidance room Rm2. The instructor 2 is able to be in the guidance room Rm2 and use the monitor 25 to confirm the endoscopic image 30.

[0045] The guidance terminal 26 is, for example, a tablet terminal with a touch panel 27. The guide 2 can operate the touch panel 27 with his finger, stylus, etc., so as to input instructions on the endoscope camera image 30.

[0046] Because of this input, Figure 3 A composite image 50, formed by overlaying the annotated images 41 and 42 representing the instructions of Instructor 2 onto the endoscopic image 30, is displayed on the monitor 14. The surgeon 1 can perform surgery on the patient 3 while confirming the instructions of Instructor 2 displayed on the monitor 14.

[0047] <2. Structure of the Auxiliary System>

[0048] Then, refer to Figure 4 Explain the structure of the auxiliary system 100. Figure 4 This is a block diagram illustrating an example of the structure of the auxiliary system 100.

[0049] The auxiliary system 100 includes a surgical side system 10 configured on the operating room Rm1 side and a guidance side system 20 configured on the guidance room Rm2 side. The surgical side system 10 and the guidance side system 20 are separate from each other and can communicate with each other via wired or wireless transmission paths.

[0050] The operating system 10 includes an endoscope 11, an image generation unit 12, an image synthesis unit 13, a monitor 14, and a data transmission unit 15.

[0051] Endoscope 11 has an imaging element, and the image signals acquired by the imaging element are converted by A / D conversion to endoscopic image data Id (which refers to the luminance value of a predetermined hue in pixels) representing the luminance of the image in pixels. Figure 2 Image data of endoscopic camera image 30 as shown (as shown in the image data).

[0052] The image generation unit 12 is composed of, for example, an image processor, and generates annotation image data Ad1 based on the drawing information PI sent from the guidance system 20.

[0053] The information PI drawn here includes coordinates, line type, color, line width, and other information specified through operational input.

[0054] Additionally, the annotated image data Ad1 is used for overlaying with the endoscopic image 30. Figure 5 The image data shown in annotation images 41 and 42 are generated based on the rendering information PI to create the annotation image data Ad1. The same applies to the annotation image data Ad2, which will be described later.

[0055] The image compositing unit 13 is implemented, for example, by a video mixer device made of hardware or a dedicated circuit mounted on an FPGA (Field Programmable Gate Array), and combines various input image data. In this example, the endoscopic camera image data Id and the annotation image data Ad1 are combined in the image compositing unit 13 to generate the surgical-side composite image data Cd1.

[0056] The monitor 14 is configured, for example, as a liquid crystal display device, and displays images on a display panel based on the provided image data. In this example, the images are displayed on the monitor 14 based on the provided endoscopic camera image data Id and the surgical side composite image data Cd1.

[0057] The data transmission unit 15 is configured to communicate with the guidance system 20, for example, and to transmit various types of data to the guidance system 20. In this example, it specifically transmits endoscopic camera image data Id and surgical-side synthesized image data Cd1 obtained from the image synthesis unit 13.

[0058] Then, the guidance system 20 is explained.

[0059] The guidance system 20 includes an annotation interface 21, a data transmission unit 22, an image generation unit 23, an image compositing unit 24, and a monitor 25. These components are integrated as a single unit, for example, as a guidance terminal 26.

[0060] The annotation interface 21 is composed of input devices such as a touch panel, touchpad, mouse, and keyboard, and a processor that generates drawing information (PI) based on the operation input from these input devices. In this embodiment, for example, the instructor 2 uses a finger, stylus, etc. Figure 1 The touch panel 27 is used to operate the interface and input is sent to the annotation interface 21.

[0061] The data transmission unit 22 is configured to communicate with the treatment system 10, for example, and to transmit various types of data to the treatment system 10. In this example, in particular, it transmits the drawing information PI obtained in the annotation interface 21.

[0062] The image generation unit 23 is composed of, for example, an image processor, and generates annotation image data Ad2 based on the drawing information PI.

[0063] The image compositing unit 24 is implemented, for example, by a video mixing device as hardware or a software mixer configured in an information processing device, and combines various provided image data. In this example, the image compositing unit 24 combines the annotation image data Ad2 with the endoscopic camera image data Id and the surgical-side composite image data Cd1, thereby generating the guidance-side composite image data Cd2.

[0064] The monitor 25 is configured, for example, as a liquid crystal display device, and displays an image on the display panel based on the provided image data. In this example, the display is based, for instance, on the synthetic image data Cd2 synthesized on the guidance side.

[0065] Furthermore, as an example of the aforementioned systems 10 and 20, it is conceivable that the image generation units 12 and 23, the image synthesis units 13 and 24, and the data transmission units 15 and 22 are constructed in hardware, but the structure of each functional unit is not limited to the above. For example, all or part of each functional unit may also be implemented by one or more hardware devices.

[0066] Alternatively, each of the functional units could be configured as a microcomputer equipped with a CPU (Central Processing Unit), ROM (Read Only Memory), ROM (Random Access Memory), etc. In this case, the CPU executes processing based on a program stored in the ROM, thereby enabling the implementation of each function.

[0067] <3. Details of the Auxiliary System>

[0068] Reference Figure 4 The auxiliary system 100 in this embodiment will be described in detail.

[0069] First, in the operating system 10, the endoscopic camera image data Id obtained by the endoscope 11 is provided to the image synthesis unit 13.

[0070] In the absence of annotation image data Ad1 provided from the image generation unit 12, the image synthesis unit 13 provides endoscope camera image data Id to the monitor 14 and the data transmission unit 15.

[0071] Endoscopic camera image data ID is provided to monitor 14, thereby displaying endoscopic camera image 30 (see reference) on monitor 14. Figure 2 Thus, the surgeon 1 can perform the surgery while observing the condition inside the patient 3's body cavity using the monitor 14.

[0072] In addition, the data transmission unit 15 will send the endoscopic camera image data ID provided by the image synthesis unit 13 to the guidance system 20.

[0073] In the guidance system 20, the image synthesis unit 24 receives endoscopic camera image data Id from the procedure side system 10 (data transmission unit 15).

[0074] In the absence of annotation image data Ad2 provided from image generation unit 23, image synthesis unit 24 provides endoscope camera image data Id to monitor 25.

[0075] By providing the endoscopic camera image data ID, the endoscopic camera image 30 is displayed on monitor 25 (refer to...). Figure 2 Thus, the instructor 2 can be in the instruction room Rm2 and share a viewpoint with the surgeon 1 in the operating room Rm1, and can observe the state inside the patient 3's body cavity and the surgeon 1's surgical procedure.

[0076] While observing the endoscopic camera image 30 displayed on the monitor 25, the instructor 2 operates the input device such as the touch panel 27, thereby inputting graphics, text, markings, etc., representing the instructions to the practitioner 1 onto the endoscopic camera image 30.

[0077] In annotation interface 21, drawing information PI corresponding to the operation input to touch panel 27 is generated. The drawing information PI obtained in annotation interface 21 is provided to image generation unit 23 and data transmission unit 22.

[0078] The data transmission unit 22 will send the drawing information PI obtained from the annotation interface 21 to the application side system 10.

[0079] Whenever data is sent to the treatment system 10, drawing information PI with a smaller communication capacity than the annotation image data Ad2 generated by the image generation unit 23 is sent, thereby reducing the communication delay when sending data from the guidance system 20 to the treatment system 10.

[0080] In the treatment system 10, the image generation unit 12 receives drawing information PI from the guidance system 20 (data transmission unit 22). Based on the received drawing information PI, the image generation unit 12 generates annotation images 41 and 42 (see reference). Figure 5 The annotation image data Ad1.

[0081] The annotation image data Ad1 generated in the image generation unit 12 is provided to the image synthesis unit 13.

[0082] The image synthesis unit 13 synthesizes the endoscopic camera image data Id provided by the endoscope 11 with the annotation image data Ad1 provided by the image generation unit 12 to generate the surgical-side composite image data Cd1. The surgical-side composite image data Cd1 is used to display the following composite image 50 (see reference). Figure 3 The image data of the composite image 50 is the annotated images 41 and 42 (see reference). Figure 5 Overlay on endoscopic image 30 (reference) Figure 2 It is formed by ).

[0083] Here, the image synthesis unit 13 is composed of a hardware video mixing device, so the delay time from the input of the endoscopic camera image data Id and the annotation image data Ad1 to the output of the synthesized image data Cd1 on the treatment side is extremely short.

[0084] Furthermore, the image compositing unit 13 can be implemented by a processor with high computing power, replacing the hardware video mixing device. This processor executes the compositing process specified by the program, thereby achieving a processing speed comparable to that of a hardware video mixing device, for example, a processing speed with a delay time of less than 30ms from input to output. In such cases, the image compositing unit 13 can also be implemented as a software mixer.

[0085] The aforementioned processing speed enables rapid compositing, thereby providing the treatment-side composite image data Cd1 generated by this compositing process to the monitor 14 without lag, resulting in minimal time delay in display on the monitor 14.

[0086] Therefore, the surgeon 1 in operating room Rm1 will not feel a time lag and can perform surgery on patient 3 while confirming the instructions of the instructor 2 in the instruction room Rm2 on monitor 14.

[0087] In particular, the image synthesis unit 13 is a hardware video mixing device or a software mixer device using a processor with extremely high processing speed. As mentioned above, if the delay time is less than 30m seconds, the operator 1 can hardly feel the time lag when viewing the image.

[0088] The data transmission unit 15 will send the treatment-side composite image data Cd1 provided by the image synthesis unit 13 to the guidance-side system 20.

[0089] On the other hand, in the guidance system 20, the image generation unit 23 generates images 41 and 42 (referencing the drawing information PI obtained from the annotation interface 21) based on the drawing information PI obtained from the annotation interface 21. Figure 5 The annotation image data Ad2 is displayed. The annotation image data Ad2 obtained in the image generation unit 23 is provided to the image synthesis unit 24.

[0090] The image compositing unit 24 combines the treatment-side composite image data Cd1 received from the treatment-side system 10 (data transmission unit 15) with the annotation image data Ad2 provided from the image generation unit 23 to generate the guidance-side composite image data Cd2. The generated guidance-side composite image data Cd2 is provided to the monitor 25.

[0091] Furthermore, when the endoscopic camera image data Id is received from the operating system 10 instead of the operating-side composite image data Cd1, the image synthesis unit 24 synthesizes the endoscopic camera image data Id with the annotation image data Ad2 to generate the guidance-side composite image data Cd2. The generated guidance-side composite image data Cd2 is then provided to the monitor 25 in the same manner as described above.

[0092] In any of the above cases, the annotation image data Ad2 is synthesized whenever the guiding side composite image data Cd2 is generated, thus displaying annotation images 41 and 42 on monitor 25 (see reference). Figure 5 ) superimposed on endoscopic image 30 (reference) Figure 2 The composite image 50 (refer to) Figure 3 ).

[0093] In this way, compared with the case where the guidance-side synthetic image data Cd2 is generated on the treatment-side system 10 based on the transmitted drawing information PI, the guidance-side synthetic image data Cd2 is generated in the guidance-side system 20, and the guidance-side synthetic image data Cd2 can be provided to the monitor 25 without delay.

[0094] Therefore, images can be displayed on the monitor 25 without causing mental burden to the instructor 2 who is operating the touch panel 27.

[0095] Furthermore, in particular, the annotation image data Ad2 is further synthesized to generate the guidance-side composite image data Cd2, which is synthesized from the annotation image data Ad1. Thus, the instructor 2 can confirm on the monitor 25 whether there is any deviation between the annotation images 41 and 42 displayed based on the annotation image data Ad2 and the annotation images 41 and 42 displayed based on the annotation image data Ad2.

[0096] In the event of a deviation, the instructor 2 can, for example, confirm it on monitor 25. Figure 6 The deviation shown in the annotated images 41 and 42.

[0097] Here, when considering the confirmation of the displayed content, the image generation unit 23 can generate annotation image data Ad2 as image data in a different display mode than the annotation image data Ad1 generated by the image generation unit 12.

[0098] For example, such as Figure 7 As shown, the annotation images 41 and 42, which are based on the annotation image data Ad2, are displayed in dashed lines, so that they can be displayed on the monitor 25 in a way that is different from the display of the annotation image 42, which is based on the annotation image data Ad1 contained in the treatment side synthetic image data Cd1 (solid line).

[0099] and, Figure 7 The dashed line along the solid line is marked that way for ease of explanation, but in reality, they are overlapping parts.

[0100] As described above, displaying both annotation images 41 and 42 based on annotation image data Ad1 and annotation images 41 and 42 based on annotation image data Ad2 allows for confirmation, as follows.

[0101] The following situations may occur: the drawing information PI sent from the guidance system 20 may be partially missing when received by the treatment system 10 due to communication obstacles, or the display coordinates of the annotation images 41, 42, etc. may deviate due to the lack of adjustment between the settings of the various systems 10 and 20, the differences in the resolution of the monitors 14 and 25, etc.

[0102] In this situation, sometimes the display coordinates may deviate from the display on monitor 25. Figure 6 Such images.

[0103] Furthermore, in the event of partial loss of information, the image generation unit 12 of the treatment system 10, which should have generated the image for display based on the received drawing information PI, will instead generate the image for display. Figure 5 The annotation image data Ad1 in the annotated images 41 and 42 shown may sometimes generate only the displayed data due to a missing portion of the drawing information PI. Figure 8 The annotation image data Ad1 of the annotation image 42 shown (hereinafter also referred to as the missing annotation image data Ad1).

[0104] At this time, in the image synthesis unit 13, the defect annotation image data Ad1 generated by the image generation unit 12 is synthesized with the endoscopic camera image data Id to generate the following surgical-side synthesized image data Cd1, which is used for display. Figure 9 The composite image 51 shown is formed by simply overlaying the annotated image 42 onto the endoscopic image 30.

[0105] Therefore, based on the provided surgical-side synthetic image data Cd1, a synthetic image 51 lacking the annotation image 41 will be displayed on the monitor 14. In this state, the input of the instructor 2 cannot be accurately reflected on the monitor 14, and the surgeon 1 will perform the surgery without being able to grasp all the instructions of the instructor 2.

[0106] On the other hand, in the guidance system 20, the treatment-side composite image data Cd1 received from the treatment-side system 10 is combined with the annotation image data Ad2 generated in the image generation unit 23 for displaying annotation images 41 and 42. Therefore, regardless of whether the annotation image data Ad1 of the treatment-side composite image data Cd1 is missing, a composite image 50 for displaying (see reference) is generated. Figure 5 The guided side synthesized image data Cd2.

[0107] As a result, the composite image 50 based on the synthetic image data Cd2 from the instructor side is displayed on the monitor 25, so the instructor 2 cannot confirm the situation where the annotation image 41 is missing in the display on the monitor 14 on the practitioner 1 side.

[0108] In order for the instructor 2 to confirm the display status of the monitor 14 on the practitioner 1's side, it is desirable to display the image based on the annotated image data Ad1 and Ad2 in a different way.

[0109] Here, the image generation unit 23 generates annotation image data Ad2 based on the drawing information PI, for example, to display annotation images 41 and 42 with dashed lines.

[0110] In addition, the image synthesis unit 24 synthesizes the annotation image data Ad2 provided by the image generation unit 23 with the treatment-side synthesized image data Cd1 received from the treatment-side system 10 to generate the guidance-side synthesized image data Cd2.

[0111] The generated synthetic image data Cd2 is provided to monitor 25, whereby it is displayed. Figure 7 The composite image 52 shown.

[0112] Thus, the instructor 2 can confirm that a portion of the annotation image 41, which is only shown in dashed lines, is not displayed on the monitor 14 of the treatment system 10.

[0113] In other words, the instructor 2 can easily compare his instructions on the monitor 25 with the content viewed by the practitioner 1 on the monitor 14.

[0114] Based on the above, by viewing the image synthesized from the guidance side's image data Cd2, Instructor 2 can see the images of both sides based on the annotation image data Ad1 and Ad2, thereby confirming whether their instructions have been correctly conveyed to Practitioner 1. If it is recognized that the image-based instructions have not been correctly conveyed, Instructor 2 can take necessary countermeasures.

[0115] Furthermore, in this embodiment, although dashed and solid lines are used to distinguish between annotation image data Ad1 and annotation image data Ad2 in the synthetic image data Cd2, as an example of the display method, the display method is not limited to this if it can distinguish between annotation image data Ad1 and Ad2 separately. It can be distinguished by different line types, colors, brightness, etc., such as line width. Alternatively, one of them can be distinguished by highlighting.

[0116] In addition, it is also possible to consider the situation where parts of the annotation image data Ad2 generated by the image generation unit 23 that are inconsistent with the annotation image data Ad1 should be displayed separately.

[0117] <4. Summary and Variations>

[0118] According to the above-described embodiment, the auxiliary system 100 has a first system (application side system 10) and a second system (guidance side system 20) that is separate from the application side system 10.

[0119] The guidance system 20 includes an information processing unit comprising a data transmission unit 22, an image generation unit 23, and an image synthesis unit 24 (see reference). Figure 4 ).

[0120] The data transmission unit 22 sends the drawing information PI, which is based on the operation input and is required to generate the annotated image data Ad1, to the operating system 10. Additionally, the image generation unit 23 generates the annotated image data Ad2 based on the drawing information PI. Furthermore, the image synthesis unit 24 synthesizes the annotated image data Ad2 generated by the image generation unit 23 with the operating-side synthesized image data Cd1 (or endoscopic camera image data Id) received from the operating system 10.

[0121] In the guidance system 20, annotated image data Ad2 is generated. The generated annotated image data Ad2 is combined with the surgical synthetic image data Cd1 (or endoscopic camera image data Id) to generate guidance synthetic image data Cd2. Thus, compared with the data generated in the surgical system 10, guidance synthetic image data Cd2 can be obtained quickly.

[0122] Therefore, the instructor 2 is not affected by communication delays and can confirm the graphics and other inputs on the monitor 25. As a result, the instructor 2 can input comfortably and accurately, and thus can give appropriate instructions to the practitioner 1.

[0123] Furthermore, according to this embodiment, the image generation unit 23 of the guidance system 20 generates annotation image data Ad2 that can be distinguished from the annotation image data Ad1 contained in the treatment-side synthetic image data Cd1.

[0124] Therefore, it is possible to distinguish between displays based on annotated image data Ad1 and displays based on annotated image data Ad2, and to confirm whether the following annotated image data Ad1 (see reference) was generated in the treatment system 10. Figure 7 The annotated image data Ad1 reflects the drawing information PI based on the input in the guidance system 20.

[0125] Therefore, by watching monitor 25, the instructor 2 can easily confirm whether his instructions are properly displayed on monitor 14 being watched by the practitioner 1.

[0126] Furthermore, according to this embodiment, the information processing device of the treatment system 10 includes an image generation unit 12, an image synthesis unit 13, and a data transmission unit 15 (see reference 15). Figure 4 ).

[0127] The image generation unit 12 receives drawing information PI based on the operation input from the guidance system 20, and generates annotation image data Ad1 based on the received drawing information PI. Additionally, the image synthesis unit 13 synthesizes the input image data (endoscopic camera image data Id) with the annotation image data Ad1 generated by the image generation unit 12 to generate surgical-side synthesized image data Cd1. Furthermore, the data transmission unit 15 transmits the surgical-side synthesized image data Cd1 (or endoscopic camera image data Id) generated by the image synthesis unit 13 to the guidance system 20.

[0128] Here, whenever the treatment-side synthetic image data Cd1 is generated in the treatment-side system 10, drawing information PI, which has a smaller communication capacity compared to the annotation image data Ad2, is received from the guidance-side system 20 instead of receiving the annotation image data Ad2.

[0129] This reduces communication latency when sending data from the guidance system 20 to the application system 10, enabling the application system 10 to generate annotation image data Ad1 based on drawing information PI without delay.

[0130] Therefore, the instructions from Instructor 2 can be displayed on Monitor 14 without delay, and Operator 1 can immediately confirm the instructions from Instructor 2. That is, Operator 1 can perform surgery on Patient 3 more safely.

[0131] Furthermore, in this embodiment, an example is described where the image data input to the image synthesis unit 13 is set as endoscopic camera image data ID. However, the input image data is not limited to camera image data such as endoscopic camera image data ID provided by the endoscope 11. Various types of image data can be considered, such as image data recorded in the memory and obtained by reading from the memory, and image data received from external computer devices.

[0132] In addition, in this embodiment, it is desirable that the delay time of the image generation unit 12 from the input of the endoscopic camera image data Id and the annotation image data Ad1 to the output of the surgical side composite image data Cd1 is less than 30ms.

[0133] Thus, the surgeon 1 can visually confirm the synthetic image 50, etc., on the monitor 14 with low latency, thereby confirming the instructions of the instructor 2 and being able to perform surgery on the patient 3 accurately and safely.

[0134] Furthermore, in this embodiment, as an example of the auxiliary system 100, a surgical assistance system is described in which the surgeon 1 can perform surgery on the patient 3 while receiving instructions from a remote instructor 2. However, the auxiliary system 100 can be widely applied to situations where a remote instructor 2 gives instructions to the surgeon 1 while visually confirming the video image on the surgical side.

[0135] For example, the assistive system 100 can be applied to various purposes such as athletes and coaches in sports guidance, lecturers and listeners in learning assistance such as education or professional training, and presenters and listeners in remote meetings.

[0136] Furthermore, in this embodiment, it is explained that... Figure 1 The example shown is an endoscope 11 used for imaging, but the imaging device is not limited to endoscope 11 as long as it can image the surgery performed on the surgeon's side.

[0137] In addition, in this embodiment, as Figure 1 One example of the guidance terminal 26 illustrates a tablet terminal with a touch panel, but the guidance terminal 26 can be implemented by various devices, such as a PC (Personal Computer) with a mouse, keyboard, etc. for operation input, a VR (Virtual Reality) with a head-mounted display and a remote control for operation input, a main console for a surgical robot, etc.

[0138] Finally, the embodiments described in this invention are merely examples, and the invention is not limited to the embodiments described above. Furthermore, the combination of all structures described in the embodiments is not necessarily essential for solving the problem. Also, the effects described in this invention are merely examples and are not limited to them; they can achieve other effects, or even a portion of the effects described in this invention.

[0139] Explanation of reference numerals in the attached figures

[0140] 1: Operator; 2: Instructor; 3: Patient; 10: Operator-side system; 11: Endoscope; 12: Image generation unit; 13: Image synthesis unit; 14: Monitor; 15: Data transmission unit; 20: Instructor-side system; 21: Annotation interface; 22: Data transmission unit; 23: Image generation unit; 24: Image synthesis unit; 25: Monitor; 100: Auxiliary system; Ad1, Ad2: Annotated image data; PI: Drawing information; Id: Endoscopic camera image data; Cd1: Operator-side synthesized image data; Cd2: Instructor-side synthesized image data; Rm1: Operating room; Rm2: Instructor's room.

Claims

1.An information processing apparatus, an assist system having a first system and a second system separate from the first system, the second system in the assist system having the information processing apparatus, in the information processing apparatus, comprising: a data transmitting section that transmits drawing information based on an operation input, which is necessary to generate first annotation image data in the first system, to the first system; an image generating section that generates second annotation image data in accordance with the drawing information; and an image synthesizing section that synthesizes the second annotation image data generated by the image generating section with reception image data received from the first system, the drawing information being less in communication capacity than the second annotation image data, the image synthesizing section receiving, as the reception image data, synthesized image data generated by synthesizing the first annotation image data generated in the first system in accordance with the drawing information with inputted captured image data, the image synthesizing section synthesizing the received synthesized image data with the second annotation image data. 2.The information processing apparatus according to claim 1, wherein the image generating section generates the second annotation image data capable of being different in display manner from the first annotation image data included in the synthesized image data. 3.An information processing apparatus, an assist system having a first system and a second system separate from the first system, the first system in the assist system having the information processing apparatus, in the information processing apparatus, comprising: an image generating section that receives drawing information based on an operation input from the second system and generates first annotation image data in accordance with the received drawing information; an image synthesizing section that synthesizes inputted captured image data with the first annotation image data generated by the image generating section to generate synthesized image data for synthesizing with second annotation image data generated in the second system in accordance with the drawing information; and a data transmitting section that transmits the synthesized image data generated by the image synthesizing section to the second system, the drawing information being less in communication capacity than the second annotation image data. 4.The information processing apparatus according to claim 3, wherein a delay time of the image synthesizing section from inputting a plurality of image data to outputting the synthesized image data is set to be less than 30 msec. 5.An assist system having a first system and a second system separate from the first system, in the assist system, the second system having an information processing apparatus comprising: a second data transmitting section that transmits drawing information based on an operation input, which is necessary to generate first annotation image data in the first system, to the first system; a second image generating section that generates second annotation image data in accordance with the drawing information; and a second image synthesizing section that synthesizes the second annotation image data generated by the second image generating section with reception image data received from the first system. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The drawing information has less communication capacity than the second annotation image data, The second image synthesizing section receives, as the received image data, synthesis image data generated by synthesizing the first annotation image data generated in the first system in accordance with the drawing information and the inputted picked-up image data, The image synthesizing section synthesizes the received synthesis image data and the second annotation image data, The information processing apparatus of the first system includes: a first image generating section that generates first annotation image data in accordance with the drawing information received from the second system; a first image synthesizing section that generates the synthesis image data by synthesizing the inputted picked-up image data and the first annotation image data generated by the first image generating section; and a first data transmitting section that transmits the synthesis image data generated by the first image synthesizing section to the second system. ​

Images