Multi-threaded video processing utilizing cloud services to detect and analyze objects
Through the combination of multi-threaded video processing system and cloud services, the problem of low real-time processing efficiency of video streams in the prior art is solved, real-time lesion detection and video annotation output during clinical surgical procedures are realized, and processing efficiency and accuracy are improved. .
Patent Information
- Application Number
- CN202280094721.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-12
- Filing Date
- 2022-10-06
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to ensure that the lesion detection results are returned in real time during clinical surgical procedures when processing video streams. The traditional sequential computing technology has a large calculation overhead and cannot meet real-time requirements.
Using a multi-threaded video processing system, cloud services are used for distributed computing, and video streams are processed through two independent processing threads. The main processing thread is responsible for display and storage, the object processing thread is responsible for image processing and lesion detection, and cloud services are used for computation intensive. , real-time output of annotated videos.
Real-time video analysis and lesion detection during clinical surgical procedures are realized, ensuring that practitioners can obtain annotated video streams in a timely manner, improving processing efficiency and accuracy, and avoiding interference with the main processing threads by stagnating the object processing thread.
Smart Images

Figure CN119032374A_ABST
Abstract
Description
Technical Field
[0001] The presently disclosed subject matter relates to detecting and analyzing objects in video feeds, and more particularly to a multi-threaded video processing system that utilizes cloud services to detect and analyze lesions and display annotated video during surgical procedures. Background Art
[0002] Colon polyps are growths of the colon lining and are very common. Colon polyps are important because they can be or may become malignant (cancer). They are also important because based on their size, number, and microscopic anatomy (histology), clinicians can predict which patients are more likely to develop more polyps and colon cancer. Summary of the invention
[0003] Aspects of the present disclosure provide a multi-threaded video processing system that utilizes cloud services to detect and analyze lesions and output annotated video in real-time during clinical surgical procedures.
[0004] A first aspect provides a system comprising a memory and a processor, the processor being connected to the memory and configured to process a video stream captured during a clinical surgical procedure according to a process comprising: receiving a current image frame from the video stream using a first processing thread and forwarding the current image frame to a display manager; forwarding the current image frame to a memory buffer; checking the memory buffer for a new image frame using a second processing thread running independently of the first processing thread; in response to detecting a new image frame in the memory buffer, retrieving the new image frame using the second processing thread and forwarding the new image frame to a cloud service, the cloud service being configured to detect lesions in image data; receiving a result from the cloud service at the second processing thread; in response to the result indicating a newly detected lesion in the new image frame, performing object size determination on the newly detected lesion using the second processing thread and forwarding the new lesion information to the first processing thread; and receiving the new lesion information at the first processing thread and graphically overlaying the new lesion information on the current image frame at the display manager.
[0005] A second aspect provides a method for processing a video stream captured during a clinical surgical procedure, the method comprising: using a first processing thread to receive a current image frame from the video stream and forwarding the current image frame to a display manager; forwarding the current image frame to a memory buffer; using a second processing thread that runs independently of the first processing thread to check for new image frames in the memory buffer; in response to detecting a new image frame in the memory buffer, using a second processing thread to retrieve the new image frame and forward the new image frame to a cloud service, the cloud service being configured to detect lesions in image data; receiving a result from the cloud service at the second processing thread; in response to the result indicating a newly detected lesion in the new image frame, using the second processing thread to perform object size determination on the newly detected lesion and forwarding the new lesion information to the first processing thread; and receiving the new lesion information at the first processing thread, and graphically overlaying the new lesion information on the current image frame at the display manager.
[0006] A third aspect provides a system comprising a memory and a processor, the processor being connected to the memory and configured to process a video stream captured during a clinical surgical procedure according to a process comprising: receiving a current image frame from the video stream using a first processing thread and forwarding the current image frame to a display manager; forwarding the current image frame to a memory buffer; checking the memory buffer for new image frames using a second processing thread running independently of the first processing thread; in response to detecting a new image frame in the memory buffer, retrieving the new image frame using a second processing thread and forwarding the new image frame to a cloud service, the cloud service being configured to process image data; receiving a first result from the cloud service at the second processing thread; performing additional image processing on the new image frame using the second processing thread to obtain a second result, and forwarding the first result and the second result to the first processing thread; and receiving the first result and the second result at the first processing thread, and graphically overlaying the first result and the second result on the current image frame at the display manager.
[0007] Additional aspects may include one or more of the following: A system and / or method wherein, prior to forwarding a current image frame to a memory buffer: determining whether a second processing thread is busy; and in response to determining that the second processing thread is busy, graphically superimposing previous lesion information onto the current image frame.
[0008] A system and / or method wherein, in response to determining that a second processing thread is not busy, a current image frame is converted to JPEG format and the current image frame is loaded into a memory buffer. In another aspect, the new image frame is encoded into Base64 format before being forwarded to a cloud service.
[0009] A system and / or method wherein, in response to a result from a cloud service not indicating a newly detected lesion in a new image frame, a first processing thread is caused to graphically overlay previous lesion information onto a current image frame. In further aspects, wherein the previous lesion information is obtained from a memory buffer.
[0010] A system and / or method wherein forwarding new lesion information to a first processing thread comprises: loading the new lesion information into a memory buffer by a second processing thread; and detecting the new lesion information in the memory buffer using the first processing thread.
[0011] A system and / or method wherein a first processing thread and a second processing thread are executed on separate CPU cores. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] These and other features of the present invention will be more readily understood through the following detailed description of various aspects of the present invention in conjunction with the accompanying drawings, in which:
[0013] Figure 1 A multi-threaded video processing platform utilizing cloud services according to an embodiment is shown.
[0014] Figure 2 Details of a multi-threaded object detection and sizing process utilizing a cloud service are shown according to an embodiment.
[0015] Figure 3 A cloud service architecture according to an embodiment is shown.
[0016] Figure 4 An image with polyp detection results is shown according to an embodiment.
[0017] Figure 5 An annotated image according to an embodiment is shown.
[0018] Figure 6 A network according to an embodiment is depicted.
[0019] Figure 7 A computer system according to an embodiment is depicted.
[0020] The accompanying drawings are not necessarily drawn to scale. The accompanying drawings are only schematic representations and are not intended to depict specific parameters of the present invention. The accompanying drawings are intended only to depict typical embodiments of the present invention and therefore should not be considered to limit the scope of the present invention. In the accompanying drawings, the same reference numerals represent the same elements. DETAILED DESCRIPTION
[0021] Various embodiments provided herein relate to a computing infrastructure for processing video feeds to detect, size, and analyze object and image data. In some aspects, the video feeds include video captured during a clinical surgical procedure (such as a colonoscopy). The video feeds are output in real time to a practitioner performing the surgical procedure using the computed information (i.e., annotations, such as lesion detection, size determination, histology, etc.).
[0022] There are various techniques for processing video feeds to provide such information. For example, U.S. Patent 10,957,043, entitled AI Systems for Detecting and Sizing Lesions (AI Systems for Detection and Sizing of Lesions), which is incorporated herein by reference, discloses machine learning techniques for detection and sizing of lesions, which can be displayed to practitioners along with video feeds during surgical procedures. However, technical challenges arise due to the computational overhead required to implement such algorithms. When processing video streams, traditional sequential computing techniques generally cannot guarantee real-time results returned to practitioners. To address this problem, the present method uses a distributed multi-threaded computing infrastructure to process and annotate videos during surgical procedures.
[0023] refer to Figure 1 , shows an exemplary computing infrastructure 10 that processes video streams 16, 16' captured, for example, during surgical procedures from different medical operating rooms 11, 11', and outputs annotated video 18, 18' back to the medical practitioner performing the surgical procedure. The video streams 16, 16' may be captured from a camera or scope used during the surgical procedure, and the resulting annotated video 18, 18' may be output on a display (such as a computer screen, etc.). Each operating room 11, 11' includes a local processing system 12, 12' (or client) that includes: (1) a main (i.e., first) processing thread 20, 20' that processes input, output, and annotation of image frames; and (2) an object (i.e., second) processing thread 22, 22' for processing image processing tasks including generating annotation information. In addition, a set of cloud services 14, for example provided by a remote cloud system (i.e., server), are provided that perform one or more complex image processing tasks, such as object detection, diagnosis, etc., for each of the local processing systems 12, 12'.
[0024] In an exemplary embodiment, each local processing system 12 ' comprises a central processing unit (CPU) having multiple cores or a single core divided into two virtual cores or threads. Each thread is created by the operating system and performs a specific set of tasks. The two threads operate simultaneously and independently of each other. In an exemplary embodiment, the main processing thread 20 provides input / output and annotation functions to the video stream 16, while the object processing thread 22 manages the cloud service 14 (which, for example, provides a first image processing task) and performs additional image processing tasks to generate annotation information. A memory buffer 24 can be deployed to buffer video frames or annotation information transmitted between the two threads 20, 22. Using this multi-threaded approach, various advantages are obtained, for example, if image processing is stalled in the object processing thread 22, it will not interfere with the ability of the main processing thread 20 to display the video to the practitioner.
[0025] The cloud service 14 includes one or more functions that provide additional levels of image processing. For example, the cloud service 14 can implement computationally intensive processing, such as object detection and diagnosis using machine learning algorithms, etc. Each cloud service 14 can be configured to perform services in an efficient manner that manages the computational overhead required for certain image processing tasks. An additional advantage of performing such specialized services on the cloud is that data from multiple operating rooms 11, 11' or sources can be collected at a single location and used for analysis purposes, for example, to continuously train associated machine learning models.
[0026] Figure 2 A detailed process flow of a computing infrastructure for annotating and outputting a video with lesion detection and sizing information is depicted. The annotated video 18 may, for example, include a graphically superimposed rectangular, elliptical, or other graphical marker showing the location of the lesion and / or textual data showing information (such as the diameter or histology of the detected lesion). In this embodiment, a main processing thread 20 receives a video stream 16 from an image source 32 (e.g., a camera, endoscope, etc.) that processes a sequence of image frames. In operation, the next (i.e., "current") image frame is input at P1, and the current image frame is forwarded to the display manager 40 at P2 (thereby reproducing the input video stream 16), where the current image frame is rendered at P3 along with previous annotation information (P6) or new annotation information (P13). A typical video stream 16 may, for example, include 32 frames per second, which are streamed to the display manager 40 through the main processing thread 20. The resulting annotated video 18 is then output to a physical display 33. In an exemplary embodiment, the frames are in a BMP (bitmap image) file format.
[0027] Processes P4 to P13 describe the functionality of adding annotations to each frame at the display manager 40. At P4, the main processing thread 20 determines whether the object processing thread 22 is busy, for example, processing a previous frame. If so, the thread 20 outputs the previous annotation information, i.e., the detection and size information determined for the previous frame at P5. This may also include retrieving the previous original annotation information from the memory buffer 24, which may, for example, include a collection of graphic data items that represent the type and attributes of a graphic object, such as a rectangle, a circle, and / or text. The drawing function in the main processing thread 20 is used to draw frames and annotations (e.g., BMP data), for example, based on the data in the buffer 24. The resulting graphic data is output to the display manager 40 (alternatively, the display manager 40 may be simply instructed to keep displaying the previous annotation information). At P6, the display manager 40 graphically overlays (or simply keeps) the previous annotation information on the current frame. Therefore, this ensures that each displayed frame has annotation information superimposed on it even if the object processing thread 22 cannot provide new annotation information for the current frame.
[0028] If the object processing thread is not busy at P4 , the current frame is converted into JPEG format at P7 and loaded into the memory buffer 24 for processing by the object processing thread 22 .
[0029] At P8, the object processing thread 22 checks to see if a new frame is available in the buffer 24. If no new frame is available, the process P8 waits until a new frame is available. If a new frame is available, the JPEG image is encoded to, for example, Base64 at P9, which is a common encoding scheme for transmitting images. The encoded image is then forwarded to an object detection service 30 in the cloud (which may include any type of server), where an object detection algorithm is applied to the image data to determine whether there is a lesion in the image. As described above, there are various image processing / machine learning techniques to perform object detection. In some embodiments, the service 30 returns an XML file containing the detection results (e.g., new lesions detected, new lesions not detected, location, etc.). Once the service 30 provides the results, the process P10 determines whether the result is a new result, that is, a new lesion is detected. If not, the process P10 instructs the main processing thread 20 to use the previous annotation information at P5 for the current frame being displayed. If at P10 the object detection service 30 detects a new result (indicating a new lesion), then at P11 object sizing is performed to provide sizing information for the lesion. Sizing of the lesion may also be performed using any known image processing techniques. The resulting information (e.g., location, size, etc.) is packaged into a new set of annotation information and loaded into the memory buffer 24. In an exemplary embodiment, the object detection information is returned as XML and converted into graphics data when loaded into the buffer 24.
[0030] At P12, the main processing thread 20 detects the new annotation information and forwards it to the display manager 40. This may also include converting the annotation data in the buffer into graphic BMP data using a drawing function. At P13, the display manager 40 graphically overlays the new annotation information onto the current frame. The current frame and these annotations are drawn simultaneously in the same thread, or the current frame is drawn together with the annotations from the previous frame.
[0031] Thus, each frame from the image source 32 is transmitted through the main processing thread 20 and output as annotated video 18 to a display 33 (e.g., a monitor in an operating room). If the object processing thread 22 is available, the input BMP frame is converted to JPEG and sent to the object processing thread 22 via the memory buffer 24, where it is encoded to Base64 and forwarded to the object detection service 30 on the cloud. The results are returned as XML and converted into a format that supports graphical annotation, and object size determination is performed. Finally, the results are returned to the main processing thread 20 via the memory buffer 24, where the new annotation information or previous annotation information is graphically overlaid with the current frame via the display manager 40.
[0032] If the object processing thread 22 is busy processing the last frame, the main processing thread 20 uses the annotation information of the previously analyzed frame from the memory buffer 24. In this case, the display manager 40 outputs the visual annotation information of the previous frame on top of the current frame, and the result is displayed on the monitor.
[0033] Figure 3 An exemplary cloud service 14 implementation is depicted, in which each of the various local processing systems 12, 12' (i.e., clients) interfaces with a router / server / queue proxy / dealer platform 50. Each client has a corresponding REP worker 52 in the cloud. In operation, the cloud server starts a set of worker threads. Each worker thread creates a REP socket and then processes requests on the socket. The service uses each REP socket to receive requests from the client and send replies to the client. This socket type allows an alternating sequence of receive calls and subsequent send calls. Each request received is fair queued among all clients, and each reply sent is routed to the client that issued the last request. The worker thread behaves like a single-threaded server. The only difference is the transport (inproc instead of tcp) and the bound connection direction.
[0034] The server also creates a ROUTER socket to talk to the clients, and binds that ROUTER socket to its external interface (e.g., via tcp). The server also creates DEALER sockets to talk to the workers, and binds those DEALER sockets to the internal interface (e.g., via inproc). The server also starts a proxy that connects the two sockets. The proxy picks up incoming requests from all clients and distributes those requests to the workers. It also routes replies to the corresponding clients.
[0035] Each client uses a REQ socket to send requests and receive replies from services running on the cloud server. This socket type only allows alternating sequences of send and subsequent receive calls. A REQ socket can be connected to n REP or ROUTER sockets. Each request sent round-robin between all connected services, and each reply is matched with the last request issued. If no services are available on the cloud server, any send operation on the socket will block until at least one service becomes available. A REQ socket will not discard any messages.
[0036] Services running on cloud servers use REP sockets to receive requests and send replies to clients. This socket type allows only alternating sequences of receive and subsequent send calls. Each request received is queued between all clients, and each reply sent is routed to the client that issued the last request. If the original requester no longer exists, the response is silently discarded.
[0037] The DEALER socket type talks to a set of anonymous peers, sending and receiving messages using a polling algorithm. DEALER serves as an asynchronous REQ replacement for clients talking to a REP or ROUTER server. Messages received by the DEALER from all connected peers are queued.
[0038] The ROUTER socket type talks to a set of peers, sending each outgoing message to a specific peer connection using explicit addressing. ROUTER serves as an asynchronous replacement for REP, and is often used as the basis for a server talking to a DEALER client.
[0039] Figure 4 An exemplary annotated frame showing the results of lesion detection is depicted. The image depicts a detected polyp within a square. Figure 5 An exemplary annotated frame showing sizing information for a detected polyp is depicted. In this case, the polyp is enclosed in an oval and is shown to be 11.6 mm in diameter.
[0040] It should be appreciated that aspects of the described video processing infrastructure may be implemented in any manner, for example, as a stand-alone system, a distributed system, within a network environment, etc. Figure 6 A non-limiting network environment 101 in which various aspects of the present disclosure may be implemented includes one or more client machines 102A to 102N installed within the computing environment 101, one or more remote machines 106A to 106N, one or more networks 104, 104', and one or more devices 108. The client machines 102A to 102N communicate with the remote machines 106A to 106N via the networks 104, 104'.
[0041] In some embodiments, client machines 102A to 102N communicate with remote machines 106A to 106N via an intermediate device 108. The illustrated device 108 is located between networks 104, 104' and may also be referred to as a network interface or gateway. In some embodiments, the device 108 may operate as an application delivery controller (ADC) to provide clients with access to business applications and other data deployed in a data center, cloud, or as software as a service (SaaS) across a range of client devices, and / or provide other functions such as load balancing. In some embodiments, multiple devices 108 may be used, and the device 108 may be deployed as part of the network 104 and / or 104'.
[0042] The client machines 102A-102N may be generally referred to as client machines 102, local machines 102, clients 102, client nodes 102, client computers 102, client devices 102, computing devices 102, endpoints 102, or endpoint nodes 102. The remote machines 106A-106N may be generally referred to as servers 106 or server farms 106. In some implementations, the client device 102 may have the ability to function both as a client node seeking to access resources provided by the server 106 and as a server 106 providing access to hosted resources to other client devices 102A-102N. The networks 104, 104' may be generally referred to as networks 104. The networks 104 may be configured in any combination of wired and wireless networks.
[0043] Server 106 can be any server type, such as: a file server; an application server; a web server; a proxy server; an appliance; a network appliance; a gateway; an application gateway; a gateway server; a virtualization server; a deployment server; a Secure Sockets Layer Virtual Private Network (SSLVPN) server; a firewall; a web server; a server executing an Active Directory; a cloud server; or a server executing an application accelerator that provides firewall functionality, application functionality, or load balancing functionality.
[0044] Server 106 may execute, operate, or otherwise provide an application, which may be any of the following: software; a program; executable instructions; a virtual machine; a hypervisor; a web browser; a web-based client; a client-to-server application; a thin-client computing client; an ActiveX control; a Java applet; software related to Voice over Internet Protocol (VoIP) communications, such as a soft IP phone; an application for streaming video and / or audio; an application for facilitating real-time data communications; an HTTP client; an FTP client; an Oscar client; a Telnet client; or any other set of executable instructions.
[0045] In some implementations, the server 106 may execute a remote presentation service program or other program that uses a thin client or remote display protocol to capture display output generated by an application executing on the server 106 and send the application display output to the client device 102 .
[0046] In other embodiments, the server 106 may execute a virtual machine that provides access to a computing environment to a user of the client device 102. The client device 102 may be a virtual machine. The virtual machine may be managed by, for example, a hypervisor, a virtual machine manager (VMM), or any other hardware virtualization technology within the server 106.
[0047] In some implementations, the network 104 may be: a local area network (LAN); a metropolitan area network (MAN); a wide area network (WAN); a primary public network 104; and a primary private network 104. Additional implementations may include a network 104 of a mobile phone network that uses various protocols to communicate between mobile devices. For short-range communications within a wireless local area network (WLAN), protocols may include 802.11, Bluetooth, and near field communication (NFC).
[0048] Elements of the described solution may be embodied in a computing system such as Figure 7The computing system shown, wherein the computing device 300 may include one or more processors 302, volatile memory 304 (e.g., RAM), non-volatile memory 308 (e.g., one or more hard disk drives (HDD) or other magnetic or optical storage media, one or more solid state drives (SSD) (such as flash drives or other solid state storage media), one or more hybrid magnetic and solid state drives, and / or one or more virtual storage volumes (such as cloud storage), or a combination of such physical storage volumes and virtual storage volumes or arrays thereof), a user interface (UI) 310, one or more communication interfaces 306, and a communication bus 312. The user interface 310 may include a graphical user interface (GUI) 320 (e.g., a touch screen, a display, etc.) and one or more input / output (I / O) devices 322 (e.g., a mouse, a keyboard, etc.). The non-volatile memory 308 stores an operating system 314, one or more applications 316, and data 318, so that computer instructions such as the operating system 314 and / or the applications 316 are executed by the processor 302 in the volatile memory 304. Data may be input using the input devices of the GUI 320 or received from the I / O devices 322. The various elements of the computer 300 may communicate via the communication bus 312. The computer 300 shown in FIG. 11 is shown as a client, server, and / or device by way of example only, and may be implemented by any computing or processing environment and utilizing any type of machine or collection of machines that may have suitable hardware and / or software capable of operating as described herein.
[0049] Processor 302 can be implemented by one or more programmable processors that execute one or more computer programs to perform the functions of the system. As used herein, the term "processor" describes an electronic circuit that performs a function, operation, or sequence of operations. Functions, operations, or sequences of operations can be hard-coded into an electronic circuit, or soft-coded by instructions kept in a memory device. A "processor" can use digital values or use analog signals to perform functions, operations, or sequences of operations. In some embodiments, a "processor" can be embodied in one or more application specific integrated circuits (ASICs), microprocessors, digital signal processors, microcontrollers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), multi-core processors, or general-purpose computers with associated memory. A "processor" can be analog, digital, or mixed signals. In some embodiments, a "processor" can be one or more physical processors or one or more "virtual" (e.g., remotely located or "cloud") processors.
[0050] Communication interface 306 may include one or more interfaces to enable computer 300 to access a computer network such as a LAN, WAN, or the Internet through various wired connections and / or wireless or cellular connections.
[0051] In the described embodiments, the first computing device 300 can execute applications on behalf of a user of a client computing device (e.g., a client); can execute a virtual machine that provides an execution session within which applications are executed on behalf of the user; or a client computing device (e.g., a client), such as a hosted desktop session, can execute a terminal services session to provide a hosted desktop environment; or can provide access to a computing environment that includes one or more of the following: one or more applications, one or more desktop applications, and one or more desktop sessions in which one or more applications can execute.
[0052] As will be understood by those skilled in the art upon reading the following disclosure, the various aspects described herein may be embodied as a system, device, method, or computer program product (e.g., a non-transitory computer-readable medium having computer-executable instructions for performing the operations or steps). Thus, these aspects may take the form of a complete hardware implementation, a complete software implementation, or an implementation combining software and hardware aspects. In addition, such aspects may take the form of a computer program product stored by one or more computer-readable storage media having a computer-readable program code or instructions embodied in or on the storage medium. Any suitable computer-readable storage medium may be utilized, including a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, and / or any combination thereof.
[0053] The foregoing figures illustrate some of the associated processes according to several embodiments of the present disclosure. In this regard, each figure or box within the flow chart of the figure represents a process associated with an embodiment of the described method. It should also be noted that in some alternative implementations, the actions indicated in the figures or boxes may not occur in the order indicated in the figures, or may actually be performed substantially simultaneously or in reverse order, for example, depending on the actions involved. In addition, those of ordinary skill in the art will recognize that additional boxes describing the processes may be added.
Claims
1. A system, comprising: Memory; as well as a processor coupled to the memory and configured to process a video stream captured during a clinical surgical procedure according to a process comprising: receiving, using a first processing thread, a current image frame from the video stream and forwarding the current image frame to a display manager; forwarding the current image frame to a memory buffer; checking the memory buffer for new image frames using a second processing thread that runs independently of the first processing thread; in response to detecting a new image frame in the memory buffer, retrieving the new image frame using the second processing thread and forwarding the new image frame to a cloud service configured to detect lesions in image data; receiving, at the second processing thread, a result from the cloud service; in response to the result indicating a newly detected lesion in the new image frame, performing object size determination on the newly detected lesion using the second processing thread and forwarding new lesion information to the first processing thread; as well as The new lesion information is received at the first processing thread, and the new lesion information is graphically overlaid on the current image frame at the display manager.
2. The system according to claim 1, wherein: Before forwarding the current image frame to the memory buffer: determining whether the second processing thread is busy; and In response to determining that the second processing thread is busy, previous lesion information is graphically superimposed onto the current image frame.
3. The system according to claim 2, wherein: In response to determining that the second processing thread is not busy, the current image frame is converted into a JPEG format and loaded into the memory buffer.
4. The system according to claim 3, wherein: Before forwarding the new image frame to the cloud service, the new image frame is encoded into Base64 format.
5. The system according to claim 1, wherein: In response to the result from the cloud service not indicating a newly detected lesion in the new image frame, causing the first processing thread to graphically overlay previous lesion information onto the current image frame.
6. The system according to claim 5, wherein: The previous lesion information is obtained from the memory buffer.
7. The system according to claim 1, wherein: Forwarding the new lesion information to the first processing thread includes: Loading the new lesion information into the memory buffer by the second processing thread; and New lesion information in the memory buffer is detected using the first processing thread.
8. The system according to claim 1, wherein: The first processing thread and the second processing thread run on separate CPU cores.
9. A method of processing a video stream captured during a clinical surgical procedure, the method comprising: receiving, using a first processing thread, a current image frame from the video stream and forwarding the current image frame to a display manager; forwarding the current image frame to a memory buffer; checking the memory buffer for new image frames using a second processing thread that runs independently of the first processing thread; in response to detecting a new image frame in the memory buffer, retrieving the new image frame using the second processing thread and forwarding the new image frame to a cloud service configured to detect lesions in image data; receiving, at the second processing thread, a result from the cloud service; in response to the result indicating a newly detected lesion in the new image frame, performing object size determination on the newly detected lesion using the second processing thread and forwarding new lesion information to the first processing thread; as well as The new lesion information is received at the first processing thread, and the new lesion information is graphically overlaid on the current image frame at the display manager.
10. The method according to claim 9, wherein: Before forwarding the current image frame to the memory buffer: determining whether the second processing thread is busy; and In response to determining that the second processing thread is busy, previous lesion information is graphically superimposed onto the current image frame.
11. The method according to claim 10, wherein: In response to determining that the second processing thread is not busy, the current image frame is converted into a JPEG format and loaded into the memory buffer.
12. The method according to claim 11, wherein: Before forwarding the new image frame to the cloud service, the new image frame is encoded into Base64 format.
13. The method according to claim 9, wherein: In response to the result from the cloud service not indicating a newly detected lesion in the new image frame, causing the first processing thread to graphically overlay previous lesion information onto the current image frame.
14. The method according to claim 13, wherein: The previous lesion information is obtained from the memory buffer.
15. The method according to claim 9, wherein: Forwarding the new lesion information to the first processing thread includes: Loading the new lesion information into the memory buffer by the second processing thread; and New lesion information in the memory buffer is detected using the first processing thread.
16. The method according to claim 9, wherein: The first processing thread and the second processing thread run on separate CPU cores.
17. A system, comprising: Memory; as well as a processor coupled to the memory and configured to process a video stream captured during a clinical surgical procedure according to a process comprising: receiving, using a first processing thread, a current image frame from the video stream and forwarding the current image frame to a display manager; forwarding the current image frame to a memory buffer; checking the memory buffer for new image frames using a second processing thread that runs independently of the first processing thread; in response to detecting a new image frame in the memory buffer, retrieving the new image frame using the second processing thread and forwarding the new image frame to a cloud service, the cloud service being configured to process image data; receiving, at the second processing thread, a first result from the cloud service; performing additional image processing on the new image frame using the second processing thread to obtain a second result, and forwarding the first result and the second result to the first processing thread; as well as The first result and the second result are received at the first processing thread, and the first result and the second result are graphically overlaid on the current image frame at the display manager.
18. The system of claim 17, wherein: The cloud service includes a diagnostic service.
19. The system of claim 17, wherein: The further image processing comprises object size determination.
20. The system of claim 17, wherein: The first processing thread and the second processing thread run on separate CPU cores.
