Medical information control system, signal processing device, and medical information control method

Abstract

This invention provides a medical information control system, a signal processing device, and a medical information control method that support more advanced surgical procedures. [Solution] In a medical information control system, the signal processing server 20 has a real-time signal processing application 100 installed in a first container virtual area, a best-effort signal processing application 300 installed in a second container virtual area, multiple virtual shared buffers 200, 400 accessible by the real-time signal processing application 100 and a best-effort signal processing application 300 installed in a second container virtual area, and a control application 500. The control application controls the real-time signal processing application or the best-effort signal processing application to alternately use the first virtual memory area, the second virtual memory area, and the third virtual memory area of ​​the shared buffer for writing to the real-time signal processing application and reading to the best-effort signal processing application.

Description

Technical Field

[0001] The present disclosure relates to a medical information control system, a signal processing device, and a medical information control method.

Background Art

[0002] Surgeries using medical imaging devices such as endoscopes and video microscopes enable more precise procedures and are expected to be supported by image processing techniques. At this time, it is desirable to suppress the image processing performed on the medical images generated by the medical imaging device from interfering with the procedure. Therefore, in image processing, for example, real-time performance such as the completion of image processing for one frame within a certain period of time and low latency such as a small processing time, that is, delay, involved in image processing are required. At the same time, it is required to display information useful for improving the efficiency of the procedure through complex image processing.

[0003] Therefore, an external device (IP converter) that performs image processing such as rotation correction on medical images has been proposed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Furthermore, it is conceivable to connect a medical imaging device to a server and perform signal processing on the server to support more advanced procedures.

Means for Solving the Problems

[0006] To solve the above problems, the present disclosure provides a surgical server connected via a network to a medical device installed in an operating room, wherein the surgical server acquires medical images generated by the medical device, executes a real-time signal processing application and a best-effort signal processing application installed in a container virtual area and selectable by the user, and superimposes the processing results of the best-effort signal processing application onto the medical image processed by the real-time signal processing application. A medical information control system will be provided.

[0007] According to this disclosure, a surgical server connected via a network to medical equipment installed in an operating room is used to acquire medical images generated by the medical equipment, to execute a real-time signal processing application and a best-effort signal processing application installed in a container virtual area and selectable by the user, and to superimpose the processing results of the best-effort signal processing application onto the medical image processed by the real-time signal processing application. A method for controlling medical information is provided. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example of the configuration of a medical information control system according to the embodiment of this disclosure. [Figure 2] A block diagram showing an example configuration of a signal processing server. [Figure 3] A diagram illustrating the configuration and processing example of a virtual shared buffer. [Figure 4] A diagram illustrating a configuration and processing example using double buffering. [Figure 5] Figure 2 shows an example of the superposition process. [Figure 6] Figure 2 shows another example of the superposition process. [Figure 7] A diagram showing an example of the processing sequence of this embodiment. [Figure 8] A diagram illustrating examples of forceps recognition and bleeding site recognition. [Figure 9] A diagram showing an example configuration of a medical information control system according to Comparative Example 2. [Figure 10] A diagram showing an example configuration of a medical information control system according to Comparative Example 3. [Figure 11] A block diagram showing an example of the configuration of a signal processing server according to a modification of the first embodiment. [Figure 12] A block diagram showing an example configuration of a signal processing server according to the second embodiment. [Figure 13] A block diagram showing an example configuration of a signal processing server according to the third embodiment. [Figure 14] A block diagram showing an example configuration of a signal processing server according to the fourth embodiment. [Figure 15] A block diagram showing an example of a GPU configuration according to the fourth embodiment. [Figure 16] A block diagram showing the configuration and processing example of a virtual shared buffer according to the fourth embodiment. [Modes for carrying out the invention]

[0009] Embodiments of a medical information control system, a signal processing device, and a medical information control method will be described below with reference to the drawings. While the following description will focus on the main components of the medical information control system, there may be components and functions not shown or described. The following description does not exclude any components or functions not shown or described.

[0010] In surgery, both real-time signal processing, where latency is critical, such as applying brightness correction to medical images acquired from a surgical endoscope, and best-effort signal processing, where results are prioritized even if it requires a large amount of computation, such as tumor identification using image recognition, are sometimes required simultaneously.

[0011] At this time, it is required to perform real-time signal processing and best-effort signal processing simultaneously in parallel. Therefore, a medical imaging device is connected to a signal processing server, and the signal processing server operates in parallel a real-time application that requires low-latency processing and a best-effort application that does not strongly require low latency but requires complex computational processing, and it is required to display information useful for improving the efficiency of the procedure while maintaining real-time performance.

[0012] However, when reflecting the processing results of a real-time application and the processing results of a best-effort application in a single display image within the signal processing server, buffer processing is required because the processing speeds of the real-time application and the best-effort application are different.

[0013] (First Embodiment) FIG. 1 is a diagram showing an example of the configuration of a medical information control system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the medical information control system 1 according to the present embodiment includes a plurality of image imaging devices 10, a signal processing server 20, a plurality of input-side IP (Internet Protocol) converters 30, an IP switch 40, a plurality of output-side IP converters 50, a plurality of image receiving devices 60, an operation terminal 70, and a control server 80. Further, the plurality of input-side IP converters 30, the IP switch 40, the plurality of output-side IP converters 50, the operation terminal 70, and the control server 80 constitute a video network 2000 as, for example, an in-hospital image network system.

[0014] The imaging device 10 functions as an image sending device. For example, the imaging device 10, also referred to as a modality, is a device used for capturing medical images. For example, when the imaging device 10 is an endoscope, it has an endoscope camera head and a CCU (Camera Control Unit). The imaging device 10 is not limited to endoscopes and may be various medical devices such as an operative field camera or a microscope. That is, the imaging device 10 is not limited to endoscopes and may be any device having a function of capturing medical images, and its configuration is not particularly limited.

[0015] In this embodiment, a set of signal values associated with coordinate information is referred to as an image. Also, in this embodiment, an image may be referred to as image data or an image signal. In particular, a set of signal values associated with coordinate information corresponding to the position information of imaging elements in an imaging unit, for example, a two-dimensional sensor, in the imaging device 10 is referred to as a medical image. That is, in the medical information control system 1, a medical image is processed as medical image data or a medical image signal.

[0016] The signal processing server 20 is a signal processing device. The signal processing server 20 is a signal processing device (computer) that executes signal processing such as image processing. The signal processing server 20 is, for example, a server device arranged outside the operating room and is connected to a plurality of IP converters 30 arranged in different operating rooms via an IP switch 40 and a video network 2000. The signal processing server 20 executes processing based on a control command output from the control server 80 in response to an operation instruction via, for example, an operation terminal 70. The details of the signal processing server 20 will be described later. Also, the signal processing server 20 according to this embodiment corresponds to an operation server.

[0017] The placement of the signal processing server 20 in the medical information control system 1 is not limited to the configuration where it is located outside the operating room as described above. For example, the operation of the signal processing server 20 may be in an on-premise configuration, where it is located inside the operating room or within the facility where the operating room is located, or it may be in a cloud configuration, where it is located outside the operating room or outside the facility where the operating room is located. In other words, the signal processing server 20 in the medical information control system 1 may be installed in any location as long as it satisfies the conditions related to the operation of the medical information control system 1. However, if the signal processing server 20 is located outside the operating room, it is possible to use large cooling equipment that cannot be installed inside the operating room, thereby enabling higher performance of the image processing server.

[0018] The IP converter 30 is a converter that converts the image signal supplied from the image imaging device 10 into an IP-transmittable signal, such as an Ethernet® signal. The input and output on the image imaging device 10 and the image receiving device 60 are typically SDI, HDMI®, and DisplayPort interfaces. In other words, the IP converter 30 converts the image signal supplied from interfaces such as SDI, HDMI, and DisplayPort into an IP-transmittable signal. The control server 80 controls which image receiving device 60 the image imaging device 10 is connected to, according to the user's operation instructions via the operation terminal 70.

[0019] The IP switch 40 (optical switcher, etc.) has the function of transmitting received signals to predetermined devices based on control commands output from the control server 80 in response to operation instructions via, for example, the operation terminal 70. As a result, the image signal supplied from the image imaging device 10 is converted into, for example, an Ethernet signal by the corresponding IP converter 30, and output by the IP switch 40 to the signal processing server 20 or IP converter 50. For example, the IP converters 30 and 50 and the IP switch 40 are connected by Ethernet or the like.

[0020] The IP converter 50 converts the IP-transmittable signal supplied from the IP switch 40 into a signal compatible with the input interface on the image receiving device 60 side, such as SDI, HDMI®, and DisplayPort, and supplies it to the corresponding image receiving device 60. More specifically, the IP converter 50 converts the supplied IP-transmittable signal into a display image signal. Then, the IP converter 50 supplies the generated display image signal to the corresponding image receiving device 60.

[0021] The image receiving device 60 is, for example, a monitor. The image receiving device 60 displays medical images based on display image signals supplied from the IP converter 50. The operation terminal 70 consists of, for example, a mouse and keyboard. The operation terminal 70 inputs operation signals to the control server 80 in response to user operations. The control server 80 is a control device (computer). The control server 80 controls the entire medical information control system 1 according to the operation signals from the operation terminal 70.

[0022] Thus, the video network 2000 is constructed as an Ethernet-based video network. An Ethernet-based video network makes it possible to reduce physical space. In this case, instead of directly connecting the image imaging device 10 and the image receiving device 60, they can be connected via the video network 2000 in the operating room or within the hospital. This allows various images used in surgery, such as those from endoscopes, ultrasound diagnostic devices, and vital signs monitors, to be displayed and switched on any monitor.

[0023] Incidentally, various image imaging devices 10 are used in operating rooms, and sometimes the desired image processing functions are not available in the medical imaging device. Therefore, in this embodiment, the signal processing server 20 performs image processing on the medical images output from the image imaging device 10, thereby realizing functions that are independent of individual devices or vendors. Furthermore, the operability of the image processing of the image imaging device 10 can also be standardized.

[0024] As described above, the medical information control system 1 is required to perform image processing on images supplied from the image imaging device 10 with minimal delay (latency) so as not to interfere with the procedure. For example, in image processing performed on medical images, it is required to display high-resolution images such as 4K on the image receiving device 60 at a frame rate such as 60fps. In this embodiment, signal processing for one frame performed within a certain time frame so as not to interfere with observation is referred to as real-time signal processing. Furthermore, these performance requirements are collectively referred to as real-time performance.

[0025] On the other hand, in addition to image processing as real-time signal processing, the practical application of medical applications that support procedures through AI-based image recognition and other means is also progressing. The processing time for these processes can vary depending on their content. Also, because they generally have a high processing load, they may operate at lower frame rates or experience fluctuations in the frame rate. On the other hand, signal processing such as recognition processing does not necessarily require real-time performance. In this embodiment, image processing that provides additional information for procedures through image recognition and the like is referred to as best-effort signal processing.

[0026] In surgery, both real-time signal processing, where latency is critical (such as applying brightness correction to medical images acquired from a surgical endoscope), and best-effort signal processing, where computationally intensive results are paramount (such as tumor identification using image recognition), are sometimes required simultaneously. In such cases, it is necessary to process both real-time and best-effort signal processing concurrently and in parallel.

[0027] Therefore, in this embodiment, the image imaging device 10 is connected to the signal processing server 20. The signal processing server 20 operates real-time applications that require low latency processing and best-effort applications that do not require low latency but require complex computational processing in parallel, displaying information that helps improve the efficiency of the procedure while maintaining real-time capabilities.

[0028] Here, the details of the signal processing server 20 will be explained based on Figure 2. Figure 2 is a block diagram showing an example configuration of the signal processing server 20. The signal processing server 20 is a server capable of running a real-time signal processing application 100 and a best-effort signal processing application 300 in parallel.

[0029] As shown in Figure 2, the signal processing server 20 includes a real-time signal processing application 100, a virtual shared buffer 200, a best-effort signal processing application 300, a virtual shared buffer 400, and a control application 500. This signal processing server 20 is a server capable of creating so-called virtual containers, and the real-time signal processing application 100, the best-effort signal processing application 300, and the control application 500 are each placed on the signal processing server as independent containers. In other words, these applications are executed on individual containers. By containerizing the real-time signal processing application 100, the best-effort signal processing application 300, and the control application 500 in this way, it becomes possible to logically separate the computing resources of both. More specifically, an example of a virtual environment that supports containers is "Docker," developed by Docker Inc.

[0030] Furthermore, in general, such virtual container environments allow you to specify the number of CPU cores used by the application, the maximum amount of memory, and the number of GPUs that can be used. By utilizing this, it is possible to prioritize the allocation of resources required for the real-time signal processing application 100 and to physically separate the computing resources of the real-time signal processing application 100 from those of the best-effort signal processing application 300.

[0031] The real-time signal processing application 100 is an application capable of processing one frame of signals within a certain time, and performs input processing 100a, distribution processing 100b, multiple signal processing 100c, 100d, superposition processing 100e, and output processing 100f, for example, by pipeline processing. Input processing 100a receives the signal supplied from the IP converter 30. Distribution processing 100b distributes the input signal supplied by input processing 100a to the virtual shared buffer 200 and signal processing 100c. Multiple signal processing 100c, 100d perform image processing such as noise reduction, various distortion corrections, resolution improvement, gradation improvement, color reproduction and color enhancement, and digital zoom. In this embodiment, signal processing includes image processing such as noise reduction, various distortion corrections, resolution improvement, gradation improvement, color reproduction and color enhancement, and digital zoom, but is not limited to these.

[0032] The superposition process 100e superimposes an image supplied from the virtual shared buffer 400 onto an image that has undergone multiple signal processing steps 100c and 100d. For example, the superposition process 100e superimposes the image using so-called alpha blending, which combines two images using a coefficient α. The output process 100f supplies the superimposed image to the IP converter 50.

[0033] The virtual shared buffer 200 is a buffer shared by the real-time signal processing application 100 and the best-effort signal processing application 300. The real-time signal processing application 100 writes the input image to the buffer, and the best-effort signal processing application 300 reads the written image.

[0034] The best-effort signal processing application 300 is an application that performs image processing to provide additional information for a procedure through image recognition, etc., and performs input processing 300a, multiple signal processing 300b, 300c, and output processing 300d, for example, by pipeline processing. Input processing 300a reads an image from the virtual shared buffer 200 and supplies it to the multiple signal processing 300b, 300c.

[0035] Multiple signal processing steps 300b and 300c are, for example, object recognition processes using SLAM (Simultaneous Localization and Mapping) or machine learning. For example, multiple signal processing steps 300b and 300c are recognition processes corresponding to forceps recognition and bleeding area recognition, and these processes are applied to the image, for example, by superimposing the area information onto a transparent image. Output processing step 300d is the process of writing the image generated by multiple signal processing steps 300b and 300c to a virtual shared buffer 400. Alternatively, multiple signal processing steps 300b and 300c may perform both forceps recognition and bleeding area recognition, or they may perform only one of these processes. Note that a transparent image is an image in which at least a portion of the region is transparent.

[0036] The virtual shared buffer 400 is a buffer shared by the real-time signal processing application 100 and the best-effort signal processing application 300. That is, the best-effort signal processing application 300 writes the processed image to the buffer, and the real-time signal processing application 100 reads the written image from the buffer. Details of the virtual shared buffer 200 and the virtual shared buffer 400 will be described later.

[0037] The control application 500 controls the real-time signal processing application 100, the virtual shared buffer 200, the best-effort signal processing application 300, and the virtual shared buffer 400. An example of the control application 500 will be described later.

[0038] Here, the configuration and processing examples of virtual shared buffer 200 and virtual shared buffer 400 will be explained based on Figure 3. Figure 3 is a diagram showing the configuration and processing example of virtual shared buffer 200. Note that the virtual shared buffer 400 has the same configuration as virtual shared buffer 200, so its explanation will be omitted.

[0039] As described above, buffering is necessary because the processing speed of the real-time signal processing application 100 and the best-effort application 300 are different. For this reason, the virtual shared buffer 200 and virtual shared buffer 400 according to this embodiment perform buffering using so-called triple buffering. As shown in Figure 3, the virtual shared buffer 200 has three virtual memory areas 200a, 200b, and 200c. Here, it is assumed that the writing side processes faster than the reading side.

[0040] As shown in the left diagram of Figure 3, the writer is writing to the first virtual memory area 200a, and the reader is reading from the third virtual memory area 200c. When the writer finishes writing to the first virtual memory area 200a, if the reading from the third virtual memory area 200c is not yet complete, the writer will write to the second virtual memory area 200b without waiting for it to finish. Similarly, the writer will alternately write to the first virtual memory area 200a and the second virtual memory area 200b until the reading from the third virtual memory area 200c is complete. Once the reading from the third virtual memory area 200c is complete, the reader will read the virtual memory area that is not currently being written to, either the first virtual memory area 200a or the second virtual memory area 200b. The case where this is the second virtual memory area 200b is shown in the right diagram of Figure 6. While the reader is accessing the second virtual memory area 200b, the writer will then alternately write to the first virtual memory area 200a and 200c. In this way, the virtual shared buffer 200 allows the write and read sides to perform processing asynchronously.

[0041] In Figure 3, if the read operation is faster than the write operation, the situation is as shown in the left diagram of Figure 3. At the time the read operation completes accessing the third virtual memory area 200c, there is a possibility that the information written to the second virtual memory area 200b is older than the information written to the third virtual memory area 200c. In such cases, the read operation can read from the third virtual memory area 200c again to transfer the data in the correct time order. As can be seen from this, the write operation can perform asynchronous processing without being affected by the processing speed of the read operation, and the read operation can always read the latest data that has been written.

[0042] In this way, by sharing asynchronous data using the virtual shared buffer 200 and virtual shared buffer 400, which are triple buffering systems, the processing speed of the best-effort application 300 can be prevented from affecting the processing speed of the real-time signal processing application 100.

[0043] Figure 4 illustrates a configuration and processing example using double buffering in virtual shared buffers 200 and 400, as Comparative Example 1. As shown in Figure 4, in so-called double buffering, data is shared between the writer and the reader using two first virtual memory areas 200a and 200b. In double buffering, the writer and the reader always access different buffers. For example, in the left diagram of Figure 4, the writer is writing data to the first virtual memory area 200a. Meanwhile, the reader is reading data from the second virtual memory area 200b. Once writing and reading are complete, the buffers being read and written are switched at the same time, as shown in the right diagram of Figure 4. If data being written is read, or data being read is written, the correct output will not be obtained, but this can be prevented by using double buffering as described above. A challenge of double buffering is that because the buffers are switched at the same time, if there is a difference in processing speed between the writer and the reader, the faster side must wait for the slower side to finish processing before proceeding, which can affect performance.

[0044] On the other hand, the real-time signal processing application 100 is expected to operate at a high frame rate of 60fps, while the best-effort signal processing application 300 is expected to operate at a slower frame rate. In other words, both need to be able to read from and write to the shared buffer asynchronously. Therefore, if the shared buffer is implemented as double buffering, in the distribution process shown in Figure 4, even if the real-time signal processing application 100 tries to write to the shared buffer, the best-effort signal processing application 300 will not have finished reading from that buffer, causing the processing of the real-time signal processing application 100 to be blocked (put into a waiting state). As can be seen from this, a configuration using double buffering for the virtual shared buffer 200 and virtual shared buffer 400 cannot guarantee real-time performance. In contrast, in the medical information control system 1 according to this embodiment, as described above, by sharing data asynchronously using triple buffering, the processing speed of the best-effort application 300 can be prevented from affecting the processing speed of the real-time signal processing application 100.

[0045] Here, we will explain the details of the superposition process 100e shown in Figure 2, based on Figure 5. Figure 5 is a diagram showing an example of the superposition process 100e shown in Figure 2. Here, we will explain an example in which a best-effort signal processing application 300 performs forceps recognition on a medical image.

[0046] For the input image 602, the signal processing 300b, 300c, etc. of the best-effort signal processing application 300 recognize the forceps using machine learning such as DL. Then, the signal processing 300b, 300c draw the bounding boxes (rectangles indicating position and size) 604a, 604b, 604c on the superimposed image 604 with transparency.

[0047] In this case, one possible approach is for the best-effort signal processing application 300 to only perform recognition, passing its position and size to the real-time signal processing application 100, which then draws the bounding box. However, this method requires drawing for each recognized forceps, making it impossible to estimate the necessary processing load. Therefore, if there are many objects to draw, it may affect real-time performance.

[0048] In contrast, in this embodiment, as shown in Figure 5, the best-effort signal processing application 300 prepares a buffer for the superimposed image 604 with transparency and performs the necessary drawing on it. The best-effort signal processing application 300 then passes the superimposed image 604 with transparency to the real-time signal processing application 100 via the virtual shared buffer 400.

[0049] In the real-time signal processing application 100, the received superimposed image 604 with transparency is superimposed on the input image 602 that has flowed through its own pipeline using alpha blending, based on the transparency information, to generate a superimposed image 602a.

[0050] In this way, the real-time signal processing application 100 does not depend on the processing performed by the best-effort signal processing application 300, and can always perform alpha blending on the entire screen. Therefore, its processing load is constant and can be easily estimated. This ensures real-time performance. In this way, the best-effort signal processing application 300 passes the superimposed image 604 with transparency to the real-time signal processing application 100, and the real-time signal processing application 100 performs only the superimposition process, thereby keeping the upper limit of the processing load of the real-time signal processing application constant.

[0051] Figure 6 shows another example of the superposition process 100e shown in Figure 2. Here, we describe an example in which the best-effort signal processing application 300 performs forceps recognition on a medical image. In this case, instead of performing alpha blending on the entire screen, the processing result of the best-effort signal processing application 300 may be superimposed on a portion of the input image 602, specifically region 6040, using picture-in-picture, as shown in Figure 6. However, the method of performing alpha blending on the entire screen may offer higher expressive power in some cases.

[0052] In this embodiment, image superposition was described, but it is also conceivable that metadata could be passed along with image data to control the execution parameters of the real-time signal processing application 100. For example, one possible collaboration would be to perform digital zoom as image processing in the real-time signal processing application 100, and then control the zoom magnification using the processing result of the best-effort signal processing application 300.

[0053] Here, the processing sequence of this embodiment will be described based on Figures 7 and 8. Figure 7 is a diagram showing an example of the processing sequence of this embodiment. Figure 8 is a diagram showing an example of forceps recognition and bleeding site recognition. In addition to forceps recognition, signal processing 300b and 300c of the best-effort signal processing application 300 perform bleeding recognition on medical images as another example of the best-effort signal processing application 300. The input image 602 is, for example, a medical image, and the best-effort signal processing application 300 uses machine learning such as DL to recognize bleeding sites on the input image 602 and draws their bounding boxes (rectangles indicating position and size) 610a and 610b on the output image 602c. In this embodiment, images may also be referred to as image data.

[0054] As shown in Figure 7, the user first starts a real-time signal processing application via the operation terminal 70 (step S100). At this time, the operation terminal 70 may communicate with the control application 500 of the signal processing server 20 via the control server 80, but here it communicates directly with the signal processing server 20.

[0055] Next, the control application 500 starts the real-time signal processing application 100 (step S102). The real-time signal processing application 100 performs image processing on the input image 602 received from the designated image imaging device 10 and outputs it to the image receiving device 60. An image of the output image at this point is shown in G10 of Figure 8. At the same time, the real-time signal processing application 100 starts reading from and writing to the virtual shared buffer 200 (step S104). However, since there is no shared application, it does not have any particular effect on the resulting image.

[0056] Next, the user, via the operating terminal 70, initiates the forceps recognition application as a best-effort signal processing application 300 for the control application 500 (step S106). Once the forceps recognition application is started via the control application 500, it performs forceps recognition on the image obtained from the virtual shared buffer 200 (step S108) and writes the result to the output virtual shared buffer 400 (step S110). There is no change in the processing content of the real-time signal processing application 100, but this results in the output image 602b shown in G20 of Figure 8. The signal processing flow at this time is equivalent to the state in Figure 5 described above, and the forceps recognition application corresponds to the best-effort signal processing application 300.

[0057] Next, the user switches the forceps recognition application to the bleeding recognition application. The user, via the operating terminal 70, performs an operation on the control application 500 to terminate the forceps recognition application (step S112). The control application 500 then terminates the forceps recognition application (step S114). At this time, the forceps recognition application clears the virtual shared buffer 400 on the output side so that no output images remain, and then terminates reading and writing (step S116).

[0058] Next, the user initiates the bleeding recognition application via the control terminal 70 to the control application 500 (step S118). Once the bleeding recognition application is started via the control application 500, it performs bleeding recognition on the image obtained from the virtual shared buffer 200 (step S120) and writes the result to the output virtual shared buffer 400 (step S122). Although there is no change in the processing content of the real-time signal processing application 100, this results in the output image 602c shown in G30 of Figure 8. The signal processing flow at this time is equivalent to the case in Figure 5 where the content of the best-effort signal processing application is changed from the forceps recognition application to the bleeding recognition application.

[0059] Subsequently, when the user exits the bleeding recognition application (steps S124, S126, S128), the output image returns to the state of G10 in Figure 8.

[0060] In this way, even if the best-effort signal processing application 300 is started, stopped, or switched, the processing content and load of the real-time signal processing application 100 remain unchanged. In other words, it simply continues the same distribution and superposition processing, so there is no image distortion or frame rate fluctuation due to switching, and the real-time performance of the real-time signal processing application 100 is guaranteed.

[0061] Figure 9 shows an example configuration of the medical information control system 1a according to Comparative Example 2. As shown in Figure 9, the medical information control system 1a according to Comparative Example 2 comprises a plurality of image imaging devices 10, a plurality of IP converters 30 on the transmitting side, an IP switch 40, a plurality of IP converters 500 on the receiving side, and a plurality of image receiving devices 60. The IP converters 500 also perform image processing.

[0062] As shown in Figure 9, the medical information control system 1a according to Comparative Example 2 does not have a signal server that performs signal processing. Therefore, in the medical information control system 1a according to Comparative Example 2, the CCU of the image imaging device 10 performs real-time signal processing and best-effort signal processing functions. As a result, in the medical information control system 1 according to Comparative Example 2, the functions within the image imaging device 10 become bloated, leading to increased development and quality control costs. In addition, providing similar functions from different equipment vendors results in inconsistent operability and poor usability.

[0063] In contrast, the medical information control system 1 according to this embodiment uses common formats such as SDI and HDMI (registered trademark) for medical images input from the image imaging device 10 to the signal processing server 20, as described above, thus enabling functions that are independent of individual devices or vendors. This allows for vendor-wide functionality and unified operability. Furthermore, since the signal processing server 20 can uniformly perform signal processing on each image imaging device 10, the functions within the image imaging device 10 can be suppressed, thereby reducing development and quality control costs.

[0064] Figure 10 shows an example configuration of the medical information control system 1b according to Comparative Example 3. As shown in Figure 9, the medical information control system 1b according to Comparative Example 3 comprises an image imaging device 10, a transmitting IP converter 30, a receiving IP converter 500, and an image receiving device 60. The IP converter 500 also performs signal processing. Specifically, the IP converter 500 performs input processing 500a, multiple signal processing 500b, 500c, and output processing 500d. As mentioned above, the image input from the image imaging device 10 to the IP converter 500 is in a common format such as SDI or HDMI (registered trademark), so it is possible to realize functions that are not dependent on individual equipment or vendors. This makes the functions common to all vendors, so the operability is also standardized.

[0065] On the other hand, the IP converter 500 is generally small and has limited computing power, which restricts the signal processing performance and functions that can be realized, such as image processing. In particular, image recognition using machine learning, which detects specific objects or situations in an image, generally has a high processing load. Furthermore, the processing load fluctuates depending on the content of the image, which impairs the real-time performance when displaying medical images output from the image imaging device 10. In contrast, the medical information control system 1 according to this embodiment performs signal processing using the signal processing server 20. In this case, as described above, triple buffering is used so that even if the best-effort signal processing application 300 is started, stopped, or switched, the processing content and load of the real-time signal processing application 100 remain unchanged. Therefore, in the real-time signal processing application 100 according to this embodiment, since the same distribution and superposition processing is continued, no image distortion or frame rate fluctuations due to switching occur, and real-time performance can be guaranteed.

[0066] As described above, according to this embodiment, the signal processing server 20 has a real-time signal processing application 100 installed in a first container virtual area, a best-effort signal processing application 300 installed in a second container virtual area, a plurality of virtual shared buffers 200, 400 accessible by both, and a control application 500. The virtual shared buffers 200, 400 each comprise a first virtual memory area 200a, a second virtual memory area 200b, and a third virtual memory area 200c, respectively. The control application 500 controls the real-time signal processing application 100 and the best-effort signal processing application 300 to alternately use the first virtual memory area 200a, the second virtual memory area 200b, and the third virtual memory area 200c for writing and reading. This allows the real-time signal processing application 100 and the best-effort signal processing application 300 to share data without interfering with each other's operations, thus maintaining the real-time nature of the real-time signal processing application 100.

[0067] Furthermore, the real-time signal processing application 100, the best-effort signal processing application 300, and the control application 500 are each placed as independent containers in a container virtual environment on the signal processing server. This allows for the logical separation of computing resources between the two applications by containerizing each application. Moreover, by allocating CPU, GPU, and other resources of the signal processing server 20 according to their priority, it becomes possible to physically separate the computing resources of each application.

[0068] (Modified version of the first embodiment) In the first embodiment of the medical information control system 1, the signal processing server 20 had only one real-time signal processing application 100. However, the modified version of the medical information control system 1 of the first embodiment differs in that the signal processing server 20 has multiple real-time signal processing applications 100. The differences from the medical information control system 1 of the first embodiment will be explained below.

[0069] Figure 11 is a block diagram showing an example configuration of a signal processing server 20 according to a modification of the first embodiment. As shown in Figure 11, the signal processing server 20 includes a plurality of real-time signal processing applications 100. With this configuration, the signal processing server can provide the real-time signal processing applications 100 to a plurality of image receiving devices 60.

[0070] Furthermore, it is possible to configure the system by combining multiple best-effort signal processing applications 300, or by combining three or more applications. For example, the signal processing server 20 can be equipped with multiple real-time signal processing applications 100 and multiple best-effort signal processing applications 300. In this way, the signal processing server 20 can be configured according to various requirements.

[0071] (Second Embodiment) The medical information control system 1 according to the second embodiment differs from the medical information control system 1 according to the first embodiment in that it is also capable of superimposed processing that takes into account the delay time by the best-effort signal processing application 300. The differences from the medical information control system 1 according to the first embodiment will be explained below.

[0072] Figure 12 is a block diagram showing an example configuration of the signal processing server 20 according to the second embodiment. As shown in Figure 12, the input image 602 distributed to the virtual shared buffer 200 has a metadata area 6020 to which a timestamp or sequential ID is added, and a difference value area 6022. Similarly, the output image 604 has a metadata area 6040a to which a timestamp or sequential ID is added.

[0073] As mentioned above, the best-effort signal processing application 300 often has a lower frame rate and more variable processing time compared to the real-time signal processing application 100. In such cases, the output of the best-effort signal processing application 300 (superimposed image) is delayed in time compared to the output of the real-time signal processing application 100. Depending on the degree of this delay, it can affect the user's operability.

[0074] In contrast, the best-effort signal processing application 300 according to this embodiment performs a time correction process that takes the above-mentioned delay into account. For example, in this embodiment, a process is performed to predict and output a future value for the delayed time. It is also possible to perform a process that displays an alert indicating that there is a delay if the delay exceeds a certain level.

[0075] The following provides a more detailed explanation. In Figure 12, the time sequence in the timestamp transfer process is shown in the order A→B→C→D→E→F→G.

[0076] In A, when the real-time signal processing application 100 receives the input image 602 from the image imaging device 10, it adds a timestamp (current time) as metadata to the metadata area 6020 of the input image 602. Then, it writes the input image 602 to the virtual shared buffer 200. It also passes the input image 602 through its own pipeline. For example, it performs a process of passing the timestamped image data from signal processing A to signal processing B, and so on.

[0077] Next, in B, the best-effort signal processing application 300 reads the timestamped input image 602 via the virtual shared buffer 200 and passes it through its own pipeline (the timestamped image data is passed from signal processing C to signal processing D to...).

[0078] Next, in C, when the best-effort signal processing application 300 outputs to the virtual shared buffer 400, it adds the timestamp value attached to the input image 602 that has flowed through the pipeline as metadata to the metadata area 6040a of the corresponding output image (superimposed image) data 604, and writes it to the virtual shared buffer 400.

[0079] Next, in D, the real-time signal processing application 100 reads the timestamped superimposed image data 6040a from the virtual shared buffer 400. It compares this with the timestamp attached to the input image 602 that came through its own pipeline. As mentioned above, since reading and writing to the virtual shared buffers 200 and 400 are performed asynchronously, the former is considered to have a past value than the latter. The real-time signal processing application 100 calculates this difference value.

[0080] Next, in E, the real-time signal processing application 100 adds the difference value calculated in D as metadata to the difference value area 6022 of the input image 602, in addition to the timestamp in A, and outputs it to the virtual shared buffer 200.

[0081] Next, in F, the best-effort signal processing application 300 reads the difference values ​​attached to the input image 602 read via the virtual shared buffer 200 and feeds them into its own pipeline, similar to B.

[0082] Next, in G, the best-effort signal processing application 300 considers the difference value assigned to the input image 602 as a delay and performs time correction for that amount during signal processing (it performs signal processing by predicting future values ​​by the amount of the difference). Alternatively, it draws an alert on the superimposed image indicating that there is a delay when outputting to the virtual shared buffer 400.

[0083] Next, the resulting image of G is superimposed in a real-time signal processing application 100 and output to the image receiving device 60 via the IP converter 50. While this embodiment assumes the use of timestamps, the concept remains the same even with sequential IDs.

[0084] As described above, the medical information control system 1 according to this embodiment measures the delay time of the best-effort signal processing application 300 relative to the real-time signal processing application 100 and performs time correction corresponding to the delay time. This makes it possible to suppress positional shifts in the output image 606 due to the delay time of the best-effort signal processing application 300 relative to the real-time signal processing application 100.

[0085] (Third embodiment) The medical information control system 1 according to the third embodiment differs from the medical information control system 1 according to the second embodiment in that it is also capable of superimposed processing that takes into account changes in the processing of the real-time signal processing application 100. The differences between the medical information control system 1 according to the third embodiment and the third embodiment will be explained below.

[0086] Figure 13 is a block diagram showing an example configuration of the signal processing server 20 according to the third embodiment. As shown in Figure 13, the input image 602 distributed to the virtual shared buffer 200 further has a metadata area 6024 to which metadata other than timestamps can also be added. The image transmission device 10 according to this embodiment corresponds to an image imaging location.

[0087] This section describes a case where image zooming is performed using signal processing 100g of the real-time signal processing application 100, and the forceps recognition application is run using the best-effort signal processing application.

[0088] First, the signal processing 100g of the real-time signal processing application 100 performs image 602 zoom processing and generates zoomed image 6028.

[0089] Next, the real-time signal processing application 100 assigns the zoom magnification of signal processing 100g to the metadata area 6024 of the input image 602. The best-effort signal processing application 300's bounding box drawing signal processing 200g reflects the magnification assigned to the metadata area 6024 when drawing the bounding box. Furthermore, the bounding box is drawn at a corresponding 1.5x zoom ratio so that when it is finally superimposed on the zoomed image, the contents stored in the buffer generated by signal processing 300a (input), 300b (forceps recognition), 300g (bounding box generation), and output 300d match the contents displayed in the zoomed image.

[0090] In this way, the real-time signal processing application 100 always passes the magnification as metadata of the virtual shared buffer 200 to the best-effort signal processing application 300. This allows the best-effort signal processing application 300 to reflect the magnification when drawing the bounding box of the forceps. Therefore, the drawing of the forceps recognition result can be synchronized with the zoom processing of the real-time signal processing application 100, and it can also follow even when zooming by continuously changing the magnification with a smooth animation. The output image 6060 is provided to the image receiving device 60, and the content of the superimposed data 6042 is displayed superimposed on the zoomed image 6028. Since the content of the superimposed data 6042 is created on a best-effort basis, it may not be updated in real time. Because the content of the superimposed data 6042 has a transparent area, it does not completely cover the underlying zoomed image 6028 in the output image 6060. Therefore, the surgical staff can perform surgery without being dissatisfied that part of the surgical site is covered by the content superimposed on the zoomed image 6028.

[0091] (Fourth Embodiment) The medical information control system 1 according to the fourth embodiment differs from the medical information control system 1 according to other embodiments in its virtual shared buffer configuration and GPU configuration. The differences from the medical information control system 1 according to the fourth embodiment will be explained below.

[0092] Figure 14 is a block diagram showing an example configuration of the signal processing server (corresponding to Server) 20 according to the fourth embodiment. Figure 14 is a block diagram similar to the configuration shown in Figure 13, except that the signal processing server includes two GPUs (Graphics Processing Units), GPU 1441 (GPU#1) and GPU 1442 (GPU#2). Furthermore, GPU#2 executes multiple best-effort signal processing applications, each having its own shared buffers 1405 and 1407, as well as a shared buffer 1403, which will be explained in more detail in Figure 16. Shared buffer corresponds to Shared Buffer, best-effort signal processing application to Best-effort application, image transmission device to Medical imaging device, IP converter to IP converter, real-time signal processing application to Real-time application, input processing to Input Processing, distribution processing to Distribution Processing, signal processing to Signal Processing, superposition processing to Superposition, output processing to Output, and image receiving device to Display.

[0093] In the configuration shown in Figure 14, GPU#1 is running a real-time signal processing application. This means that it processes images as they are received from the image transmission device for each frame and outputs frames at virtually the same rate as they are input, within a fixed time frame (a small processing delay that is generally imperceptible to the operator). An example input frame 1410 is shown as an example of being processed by multiple signal processing processes, indicated as signal processing A and signal processing B, but more processes may be running in real time on GPU#1. The zoom application in the third embodiment is just one example of a type of signal processing process that may be applied to medical images, and other applications may be applied.

[0094] The output from shared buffer 1405 provides overlay content 1430 generated by best-effort signal processing application #2, which includes signal processing C and signal processing D, executed on GPU #2. In this example, the overlay content 1430 consists of bounding boxes 1431, 1432, and 1433 for medical device / tool ​​detection. Although not necessarily generated in real time, the medical device / tool ​​detection bounding boxes 1431, 1432, and 1433 identify medical devices present in the surgical image, as shown in output image 1450. Similarly, best-effort signal processing application #1 is executed on GPU #2 by applying signal processing C and signal processing D to the input image via shared buffer 1403. In this example, best-effort signal processing application #1 detects areas in the surgical scene where the tissue shows active bleeding. At this time, the two bounding boxes 1421 and 1422 are part of the overlay content 1420 provided by the best-effort signal processing application #1, and are ultimately overlaid on image 1410 by superposition processing so as to identify bleeding sites in the output image 1450. The output image 1450 includes bounding boxes for overlays (each may be opaque but have transparency) generated by the best-effort signal processing application running on GPU #2 to identify image features in the real-time image provided by the real-time signal processing application running on GPU #1.

[0095] In Figure 14, the shared buffer 1403 is shown as being hosted on GPU#2, rather than on GPU#1. While it is possible to host the shared buffer 1403 on GPU#1, the inventors found that such a configuration would require transferring twice the amount of data between GPUs, placing a heavy load on GPU#1, which is already responsible for real-time processing. In contrast, by hosting the shared buffer on GPU#2, the amount of data that needs to be transmitted via the GPU bus 1501 (Figure 15) is halved. Note that while the example shows GPU#2 running two best-effort signal processing applications, GPU#2 can run many more best-effort signal processing applications. Even if N applications are running on GPU#2, as long as the shared buffer 1403 is hosted on GPU#2, the data bus usage and the load on GPU#1's inter-processor communication will not increase further, enabling scalability to support more best-effort signal processing applications on GPU#2.

[0096] Figure 16 is a block diagram showing the configuration and processing example of a shared buffer according to the fourth embodiment. The structure of the shared buffer 1403 is shown as having four internal dual-port shared buffers (Buffer1, Buffer2, Buffer3, Buffer4), but this structure is merely illustrative. More generally, the shared buffer 1403 is an N+2 buffer, where N is the number of "consumer processes" (or best-effort signal processing applications) supplied by the shared buffer 1403. More than N+2 internal buffers may be used, but with two or more additional buffers, it is preferable that at least one of the internal buffers is not used by either an input (a producer process such as image input from an image output device provided via distribution processing hosted on GPU#1) or a consumer process. The reason for needing two (or more) additional buffers will be explained later with respect to Figure 16.

[0097] While the input side is storing data in a specific shared buffer (for example, storing data D1 in shared buffer 1), the data in buffer 1 is unstable and incomplete, and therefore cannot be retrieved. However, once data D1 is completely written to buffer 1, consumer processes 1 and 2 (CP1 and CP2) can retrieve it. Furthermore, once the producer process has finished writing data D1 to buffer 1, shared buffer 1403 can indicate that buffer 1 is stable through any method such as controller monitoring or flag setting.

[0098] When data D1 is stored in buffer 1, the producer process begins writing data D2 to buffer 2 (or another N+1 buffer). Buffer 1 then becomes accessible to CP1 and CP2. Once data D2 is written to buffer 2, CP1 and CP2 also become accessible. However, because CP1 has a faster throughput than CP2, CP2 may access buffer 2 after CP1 has started accessing it. Thus, CP1 and CP2 may not access buffer 2 synchronously, resulting in N best-effort signal processing applications accessing N buffers at any given time. However, the producer process cannot simultaneously store data in any of these N buffers. Therefore, it activates additional buffers to store data, creating a total of N+1 buffers. However, when the producer process finishes storing data in one internal buffer, it needs to know which buffer is next available and immediately fill it with incoming data to prevent an overload situation. Furthermore, the producer process needs to know which buffer is next available, in addition to the buffer it is currently filling. Therefore, the input side (i.e., the producer process) requires two buffers that operate independently (without requiring coordination with CP1 or CP2). That is, two additional internal buffers are needed to support the producer process writing data to the shared buffer 1403 in order to support independent asynchronous input and output operations, resulting in N+2 buffers for N best-effort applications. In this case, CP2 and CP2 may be applications that do not require all input data. For example, CP2 may be a processor-intensive process that does not need to analyze all image data. In this case, CP2 may access the data in the internal buffers at a much slower rate than CP1, and such a CP2 may only need to read the second / third / fourth... data entries of buffer 1403.

[0099] Furthermore, CP1 and / or CP2 may operate in standby mode, especially if CP1 and CP2 begin reading at the same speed or at the same time. In other words, suppose it is desirable for CP1 to process the same data as CP2, but CP2 reads the data slower than CP1. In this case, CP1 can be put into standby mode to wait for CP2 to finish reading the previous buffer, and then CP1 can read the next buffer together with CP2. CP1 can also be put into standby mode if writing to the next buffer is not yet complete. The advantage of CP1 operating in standby mode is that CP1 may have lower latency requirements than CP2, allowing the consumer process with the shortest latency (the delay time from input to output) to be set to standby mode.

[0100] A shared buffer can have N+2 or more buffers, and any additional buffers beyond N+2 may be kept as reserves in case an additional process (e.g., CP3) comes online. For example, the producer process will not write to the 5th internal buffer until CP3 comes online (5 = N(now 3) + 2). Similarly, if CP goes offline, the internal buffer will also go offline and be kept as a reserve.

[0101] Furthermore, this technology can be configured as follows. (1) A surgical server connected to the medical equipment in the operating room via a network, with a container virtual environment built on it, Equipped with, The aforementioned surgical server is Multiple virtual shared buffers accessible to a first medical application installed in a first container virtual area, which performs a first process based on information output from the medical device to generate first medical information, and a second medical application installed in a second container virtual area, which performs a second process with a different processing load than the first process to generate second medical information, It has a control application, Each of the aforementioned multiple virtual shared buffers comprises a first virtual memory area, a second virtual memory area, and a third virtual memory area. The control application controls the first medical application or the second medical application to alternately use the first virtual memory area, the second virtual memory area, and the third virtual memory area for writing to the first medical application and reading to the second medical application. Medical information control system. (2) The second medical application has a larger processing load than the first medical application. (1) The medical information control system described above. (3) The medical information control system described in (1), wherein the second medical application is signal processing in which the amount of processing varies depending on the content of the input image to be processed. (4) The medical information control system according to (3), wherein the first medical application supplies an input image to the second medical application via the input-side virtual shared buffer among the plurality of virtual shared buffers, and the second medical application generates second medical information based on the input image and supplies it to the second medical application via the output-side virtual shared buffer among the plurality of virtual shared buffers. (5) The medical information control system according to (4), wherein the second medical application generates first medical information in a time series based on input images supplied in a time series, and outputs the second medical information supplied via the virtual shared buffer on the output side, with added information. (6) The medical information control system according to (4) or (5), wherein the second medical application generates the second medical information by predictive processing according to the processing time. (7) Based on the start time and end time of the processing associated with the input image, the delay time of the second medical application is measured. The second medical application is the medical information control system according to (6), which performs the predictive processing based on the delay time. (8) The medical information control system according to any one of (4) to (7), wherein the second medical application generates information indicating positional information generated by signal processing of the input image on a transparent image, and the second medical application superimposes the transparent image supplied via the output-side virtual shared buffer and the image based on the input image using the transparency. (9) The medical information control system according to (8), wherein the signal processing for the input image is the recognition of at least one of the forceps and the bleeding area. (10) The medical information control system according to any one of (4) to (9), wherein the first medical application supplies to the second medical application information relating to the input image information relating to the change in signal processing in the first medical application, and the second medical application changes the signal processing based on the information relating to the change in signal processing. (11) Medical imaging equipment and An input-side IP converter that converts medical images captured by the aforementioned medical imaging device into a predetermined image format, An IP switch for switching the supply destination of medical images of a predetermined image format and processed images supplied from the surgical server, comprising an IP switch capable of supplying medical images of the predetermined image format to the surgical server, An output-side IP converter that converts the image supplied from the IP switch into a predetermined image format, An image receiving device that displays the image supplied from the output side IP converter, A medical information control system according to any one of (1) to (10), further comprising: (12) The medical information control system according to (11), wherein at least one of the IP converter, the IP switch, and the surgical server performs processing based on control commands obtained from other devices. (13) A surgical signal processing device connected to medical equipment in an operating room via a network, in which a container virtual environment is constructed, Multiple virtual shared buffers accessible to a first medical application installed in a first container virtual area, which performs a first process based on information output from the medical device to generate first medical information, and a second medical application installed in a second container virtual area, which performs a second process with a different processing load than the first process to generate second medical information, It has a control application, Each of the aforementioned multiple virtual shared buffers comprises a first virtual memory area, a second virtual memory area, and a third virtual memory area. The control application controls the first medical application or the second medical application to alternately use the first virtual memory area, the second virtual memory area, and the third virtual memory area for writing to the first medical application and reading to the second medical application. Signal processing device. (14) A container virtual environment is built, connected to the medical equipment in the operating room via a network. Multiple virtual shared buffers accessible to a first medical application installed in a first container virtual area, which performs a first process based on information output from the medical device to generate first medical information, and a second medical application installed in a second container virtual area, which performs a second process with a different processing load than the first process to generate second medical information, It has a control application, The aforementioned plurality of virtual shared buffers are a signal processing device comprising a first virtual memory area, a second virtual memory area, and a third virtual memory area, respectively, and the method for controlling medical information of the signal processing device is as follows: A medical information control method in which the control application controls the first medical application or the second medical application to alternately use the first virtual memory area, the second virtual memory area, and the third virtual memory area for writing to the first medical application and reading to the second medical application. The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents. [Explanation of symbols]

[0102] 1: Medical information control system, 10: Medical imaging device, 20: Signal processing server, 30: IP converter, 40: IP switch, 50: IP converter, 50: Real-time signal processing application, 200: Virtual shared buffer, 300: Best-effort signal processing application, 400: Virtual shared buffer, 500: Control application.

Claims

1. An information processing system comprising a medical device and at least one signal processing server connected via a network, The aforementioned at least one signal processing server, The medical device acquires a medical image, and simultaneously performs real-time signal processing on the medical image, which involves performing a first signal processing operation on the medical image for one frame within a certain time period, and best-effort signal processing, which involves performing a second signal processing operation with a different processing amount than the first signal processing operation to generate second medical information, in parallel. An information processing system that superimposes the medical image processed by the real-time signal processing method with the processing result of the best-effort signal processing method.

2. The real-time signal processing performs the first signal processing on the medical image by first pipeline processing to generate a first image. The best-effort signal processing performs the second signal processing on the medical image by second pipeline processing to generate a second image of a size corresponding to the first image. The information processing system according to claim 1, wherein the first image and the second image are superimposed.

3. The aforementioned signal processing server, Multiple virtual shared buffers accessible to the real-time signal processing and the best-effort signal processing, It has a control application, Each of the aforementioned multiple virtual shared buffers comprises a first virtual memory area, a second virtual memory area, and a third virtual memory area. The information processing system according to claim 1, wherein the control application controls the real-time signal processing or the best-effort signal processing to alternately use the first virtual memory area, the second virtual memory area, and the third virtual memory area for writing the real-time signal processing and reading the best-effort signal processing.

4. The information processing system according to claim 1, wherein the best-effort signal processing has a larger processing capacity than the real-time signal processing.

5. The information processing system according to claim 1, wherein the best-effort signal processing is signal processing in which the amount of processing varies depending on the content of the input image to be processed.

6. The real-time signal processing unit supplies the input image to the best-effort signal processing unit via the input-side virtual shared buffer among the plurality of virtual shared buffers. The information processing system according to claim 3, wherein the best-effort signal processing generates the second medical information based on the input image and supplies it to the real-time signal processing via the output virtual shared buffer among the plurality of virtual shared buffers.

7. The information processing system according to claim 6, wherein the real-time signal processing generates first medical information in a time series based on input images supplied in a time series, and outputs the information based on the second medical information supplied via the virtual shared buffer on the output side.

8. The information processing system according to claim 7, wherein the best-effort signal processing generates the second medical information by predictive processing according to the processing time.

9. Based on the start time and end time of the processing associated with the input image, the delay time of the best-effort signal processing is measured. The best-effort signal processing is performed based on the delay time, as described in claim 8.

10. The information processing system according to claim 6, wherein the best-effort signal processing generates information indicating position information generated by signal processing on the input image on a transparent image, and the real-time signal processing superimposes the transparent image supplied via the output-side virtual shared buffer and an image based on the input image using the transparency.

11. The information processing system according to claim 10, wherein the signal processing for the input image is the recognition of at least one of the forceps and the bleeding area.

12. The information processing system according to claim 6, wherein the real-time signal processing provides information regarding changes to the signal processing in the real-time signal processing to the best-effort signal processing in association with the input image, and the best-effort signal processing modifies the signal processing based on the information regarding changes to the signal processing.

13. Medical imaging equipment and An input-side IP converter that converts medical images captured by the aforementioned medical imaging device into a predetermined image format, An IP switch for switching the supply destination of medical images of a predetermined image format and processed images supplied from the signal processing server, comprising an IP switch capable of supplying medical images of the predetermined image format to the signal processing server, An output-side IP converter that converts the image supplied from the IP switch into a predetermined image format, An image receiving device that displays the image supplied from the output side IP converter, The information processing system according to claim 3, further comprising the above.

14. The information processing system according to claim 13, wherein at least one of the IP converter, the IP switch, and the signal processing server performs processing based on control commands obtained from other devices.

15. Medical devices and, A signal processing server acquires medical images generated by the aforementioned medical device, performs real-time signal processing that executes signal processing for one frame within a certain time period, and performs N best-effort signal processing operations, and superimposes the medical image processed by the real-time signal processing with the processing results of the N best-effort signal processing operations. Equipped with, The aforementioned signal processing server, A medical information control system in which the real-time signal processing unit writes to a shared buffer, and the N best-effort signal processing units read from it, is configured such that the number of virtual shared buffer areas is N+2 or greater.

16. The aforementioned signal processing server, Set a standby mode in which the reading of one of the multiple best-effort signal processing methods is put into standby mode. The information processing system according to claim 3.

17. The aforementioned signal processing server, Based on the number of best-effort signal processing operations performed, the activation of the virtual shared buffer area is controlled. The information processing system according to claim 16.

18. The system includes multiple GPUs, and distinguishes between the GPU that performs real-time signal processing and the GPU that performs best-effort signal processing. The information processing system according to claim 1.

19. The information processing system according to claim 1, wherein the second signal processing includes AI-based image recognition processing.

20. A signal processing device connected to a device via a network, The device acquires an image generated by the aforementioned device and simultaneously performs real-time signal processing, which involves performing a first signal processing operation on the image for one frame within a certain time period, and best-effort signal processing, which involves performing a second signal processing operation with a different processing amount than the first signal processing operation to generate second information, in parallel. A signal processing device that superimposes the image processed by the real-time signal processing with the processing result of the best-effort signal processing.

21. A method for operating a medical information processing system connected to a medical device via a network, The medical image generated by the aforementioned medical device is acquired, Real-time signal processing, which performs a first signal processing for one frame of the medical image within a certain time frame, and best-effort signal processing, which performs a second signal processing with a different processing amount than the first signal processing to generate second medical information, are executed in parallel and simultaneously. The medical image processed by the real-time signal processing and the processing result of the best-effort signal processing are superimposed. How a medical information processing system operates.

Images