SYSTEM AND METHOD FOR CAPTURE DATA FROM A MEDICAL DEVICE - Patent application

Abstract

A system and method for transferring data from a medical device to a server is provided. [Solution] A method according to the remote link architecture 700 includes the steps of receiving a video stream from a medical device, capturing an image from the video stream that represents and / or depicts data over a period of time, transmitting the image to a server via a data network, extracting data from the image, sending a signal from a data module receiving the video stream from the medical device to a router indicating that the data module is connected to the network, sending a command to the data module to begin capturing the image, transferring the image to the router, broadcasting a signal indicating that the data module has captured the image, receiving the broadcasted signal at the server, and storing the image at the server.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 523,890, filed June 23, 2017, the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THEINVENTION The present disclosure relates to systems and methods for implementing a cloud-based architecture with a portal to control video streaming, data uploads, and notification destinations from medical equipment to client devices using optical character recognition and digital signal processing, with scalability to an indefinite number of users. [Background technology]

[0003] background Medical devices that monitor patients generate large amounts of data because they operate continuously. Many of these medical devices have graphic displays to deliver the data they are monitoring, but medical personnel must be physically near the medical device to view the data from the graphic display. In addition, the information on the graphic display often includes waveforms and other data presented in plots.

[0004] However, there has been a long-standing need to remotely access these medical devices so that medical personnel can monitor patients without being present, and there is also a need to provide a system that captures information from the graphic displays, extracts information from the displayed waveforms and plots, and distributes that information remotely to medical personnel. Summary of the Invention

[0005] The present disclosure relates to a method for transferring data from a medical device to a server. The method includes receiving a data video stream from the medical device. The method further includes capturing a first image from the video stream at a first time point. The first image may represent and / or represent first medical data over a first time period. The method further includes transmitting the first image to the server via a data network. The method also includes extracting the first medical data from the first image at the server.

[0006] According to one embodiment, the method further includes capturing a second image from the video stream at a second time. The second image may represent and / or represent second medical data over a second time period. The method further includes transmitting the second image to a server via a data network. The method also includes extracting the second medical data from the second image at the server.

[0007] In some embodiments, the second period begins after the first period. In other embodiments, the first period and the second period have the same duration. In certain embodiments, the duration is equal to or greater than approximately one of 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, and 1 minute. According to some embodiments, the first period and the second period have different durations.

[0008] In some embodiments, extracting the first and second medical data from the first and second images includes optical character recognition (OCR) of the first and second images.

[0009] According to some embodiments, the method further includes sensing via at least one of a pressure sensor, a temperature sensor, a flow sensor, a voltage sensor, a current sensor, an optical sensor, and an audio sensor. In some embodiments, the first and second medical data include pressure, flow, pump speed, temperature, voltage, current, and biometric status.

[0010] In other embodiments, the method further comprises repeating the steps of the features of claims 1 and 2. According to some embodiments, the server is a web server. In some embodiments, the medical device is an intravascular blood pump.

[0011] In a particular embodiment, the method further includes sending a first signal from the data module receiving the video stream from the medical device to the router indicating that the data module is connected to the network. The method further includes sending a command from the router to the data module to initiate capture of the first image. The method further includes forwarding the first image from the data module to the router. The method also includes broadcasting a second signal from the router indicating that the data module has captured the first image. The method further includes receiving, at the server, the broadcasted second signal from the router. The method further includes storing the first image at the web server.

[0012] According to some embodiments, the method further includes connecting the data module to a data network.

[0013] A second aspect of the disclosure relates to a system including a data module, a router, a client device, and a server. The data module may be configured to receive data from a medical device. The router may be communicatively coupled to the data module and configured to receive data from the data module and store the data in the storage. The client device may be configured to display the data. The server may be communicatively coupled to the client device and the router and configured to receive requests to access data from the client device, receive data from the storage, and transmit data to the client device.

[0014] A third aspect of the present disclosure relates to a method for extracting data from an image. The method includes receiving a first image. The first image may represent and / or represent data from a medical device. Additionally, the method includes masking a first portion of the first image. The first image may include the first portion and a second portion of the first image. The method further includes generating a second image consisting of the second portion of the first image. The method also includes extracting data from the second image using optical image recognition.

[0015] According to one embodiment, the first image is captured from a video stream from a medical device.

[0016] In some embodiments, the method further includes extracting first data from a first portion of the second image using optical character recognition. Additionally, the method includes determining validity of the first data by comparing the first data to reference data. The method also includes, in response to determining that the first data is valid, extracting second data from a second portion of the second image using optical character recognition.

[0017] According to some embodiments, the method further includes, in response to determining that the first data is not valid, broadcasting a signal indicating that the first data is not valid. Further, the method includes receiving a third image. The third image may be representative and / or representative of the data from the medical device.

[0018] In certain embodiments, masking the first portion of the first image includes selecting an image mask based on a size of the first image and using the image mask to mask the first portion of the first image.

[0019] In some embodiments, the method further comprises extracting waveform data from a waveform in the second image using digital signal processing.

[0020] According to some embodiments, the method further includes selecting a first pixel from the second image. The first pixel may represent and / or represent a color. Additionally, the method includes determining a color of the first pixel. The color of the first pixel may correspond to an alarm severity. The method also includes determining the alarm severity based on the color of the first pixel. In certain embodiments, the method further includes selecting a second pixel from the second image and determining the alarm severity based on the first pixel and the second pixel. [The present invention 1001] receiving a video stream from a medical device; capturing a first image from the video stream at a first time, the first image representing first data over a first time period; transmitting the first image to a server over a data network; and Extracting, at the server, the first data from the first image. 16. A method for transferring data from a medical device to a server, comprising: [The present invention 1002] capturing a second image from the video stream at a second time, the second image representing second data over a second time period; transmitting the second image to the server over a data network; and Extracting, at the server, the second data from the second image. The method of the present invention 1001 further comprises: [The present invention 1003] The method of claim 1002, wherein the second period of time begins after the first period of time. [The present invention 1004] The method of any one of claims 10 to 13, wherein said first period of time and said second period of time have the same duration. [The present invention 1005] The method of the present invention 1004, wherein said duration is equal to or greater than approximately one of 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, and 1 minute. [The present invention 1006] The method of any one of claims 10 to 12, wherein said first period of time and said second period of time have different durations. [The present invention 1007] Any of the methods of the present invention, wherein the step of extracting the first data from the first image comprises optical character recognition (OCR) of the first image. [The present invention 1008] The method of any of the preceding claims, wherein the first and second data include at least one of pressure, flow, pump speed, temperature, voltage, current, and biometric condition. [The present invention 1009] The method of the present invention 1008 further comprising sensing the first and second data via at least one of a pressure sensor, a temperature sensor, a flow sensor, a voltage sensor, a current sensor, a light sensor, and an audio sensor. [The present invention 1010] Any of the methods of the preceding inventions, further comprising the step of repeatedly performing features of invention 1001 and invention 1002. [The present invention 1011] Any of the preceding methods of the present invention, wherein the server is a web server. [The present invention 1012] Any of the methods of the preceding claims, wherein the medical device is an intravascular blood pump. [The present invention 1013] sending a first signal from a data module receiving the video stream from the medical device to a router indicating that the data module is connected to the network; sending a command from the router to the data module to initiate capture of the first image; transferring the first image from the data module to the router; broadcasting a second signal from the router indicating that the data module has captured the first image; receiving, at the server, the broadcasted second signal from the router; and storing the first image in the server. Any of the methods of the present invention further comprising: [The present invention 1014] The method of claim 1013 further comprising the step of connecting said data module to said data network. [The present invention 1015] A system configured to carry out any one of the methods of the present invention 1001 to 1014. [The present invention 1016] a data module configured to receive data from the medical device; receiving the data from the data module; The data is stored in a storage device. a router communicatively coupled to the data module, the router configured to a client device configured to display the data; receiving a request from the client device to access the data; receiving the data from storage; Transmitting the data to the client device a server communicatively coupled to the client device and the router, configured to A system including: [The present invention 1017] receiving a first image representative of data from the medical device; masking a first plurality of portions of the first image, the first image including the first plurality of portions of the first image and a second plurality of portions of the first image; generating a second image consisting of the second plurality of portions of the first image; and Extracting data from the second image using optical character recognition. 16. A method for extracting data from an image, comprising: [The present invention 1018] The method of claim 1017, wherein the first image is captured from a video stream from the medical device. [The present invention 1019] extracting data from the second image, extracting first data from a first portion of the second image using optical character recognition; determining the validity of the first data by comparing the first data to reference data; and In response to determining that the first data is valid, extracting second data from a second portion of the second image using optical character recognition. The method of any of claims 1017 and 1018, further comprising: [The present invention 1020] in response to determining that the first data is not valid, broadcasting a signal indicating that the first data is not valid; and receiving a third image representative of data from the medical device. The method of the present invention 1019 further comprises: [The present invention 1021] Masking the first plurality of portions of the first image comprises: selecting an image mask based on a size of the first image; and Masking the first plurality of portions of the first image using the image mask. Any of the methods of the present invention 1017 to 1020, comprising: [The present invention 1022] The method of any one of claims 1017 to 1021, further comprising the step of extracting waveform data from a waveform in said second image using digital signal processing. [The present invention 1023] selecting a first pixel from the second image that includes a color; determining the color of the first pixel, the color corresponding to an alarm severity; and determining the alarm severity based on the color of the first pixel. Any of the methods of the present invention 1017 to 1022, further comprising: [The present invention 1024] selecting a second pixel from the second image; and determining the alarm severity based on the first pixel and the second pixel; The method of the present invention 1023 further comprises: [Brief description of the drawings]

[0021] These and other objects and advantages will become apparent from the following detailed description considered in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout.

[0022] [Figure 1] FIG. 1 is a schematic diagram illustrating a remote link architecture, arranged in accordance with one or more aspects of the present disclosure. [Diagram 2] 1 is a flow diagram of method steps for transferring data from a medical device to a server according to an aspect of the present disclosure. [Diagram 3] 1 is a flow diagram of method steps for transferring data from a medical device to a server according to an aspect of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating a remote link architecture, arranged in accordance with one or more aspects of the present disclosure. [Diagram 5] 4 is a flow diagram of method steps for initializing a remote link according to one aspect of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram illustrating a remote link architecture, arranged in accordance with one or more aspects of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram illustrating a remote link architecture, arranged in accordance with one or more aspects of the present disclosure. [Figure 8] 1 is a schematic diagram illustrating a medical device configured in accordance with one or more aspects of the present disclosure. [Figure 9] FIG. 1 illustrates an example medical device controller configured in accordance with one or more aspects of the present disclosure. [Figure 10] FIG. 1 illustrates an example image displayed on a medical device controller screen, arranged in accordance with one or more aspects of the present disclosure. [Figure 11] 11 illustrates the example image of FIG. 10 after removing a selected portion of the image, arranged in accordance with one or more aspects of the present disclosure. [Figure 12] 12 illustrates an exemplary image of the remaining portion of the image of FIG. 11, arranged in accordance with one or more aspects of the present disclosure. [Figure 13] 13 and 14 are flow diagrams of method steps for extracting data from an image and determining the validity of the extracted data according to one aspect of the present disclosure. [Figure 14] 13 and 14 are flow diagrams of method steps for extracting data from an image and determining the validity of the extracted data according to one aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Detailed Description 1 is a schematic diagram of a remote link architecture 100. The remote link architecture 100 includes a remote link 102, a client device 104, a remote link router (RLR) 150, a web load balancer 108, a video load balancer 110, a web server 114, a video server 116, a random access memory (RAM) data type storage 118, a document data type storage 120, and a web socket server 122.

[0024] The remote link 102 may be embedded in a medical device monitoring a patient in a hospital, clinic, the patient's home, or another location. The remote link 102 captures images and distributes video streams from the medical device display and transmits the images and video to a remote link router 150. The remote link architecture 100 may include multiple remote links 102. The remote links 102 communicate with other parts of the remote link architecture 100 through the RLR 150. The RLR 150 includes an RLR load balancer 106 and an RLR server 112. The RLR 150 may include multiple RLR servers 112. The RLR server 112 may include a custom protocol used to communicate with one or more remote links 102. The RLR load balancer 106 manages the load on one or more RLR servers 112. The RLR load balancer 106 may generate priorities for the multiple remote links 102. The priorities may be based on preferences obtained from the client device 104. In another aspect, the priority is based on preferences obtained from the remote link 102. In another aspect, the priority is based on preferences obtained from the RLR server 112.

[0025] The client device 104 may be a personal computer, tablet, or mobile device with an Internet connection. A medical practitioner using the client device 104 may want to obtain information from one or more remote links 102. Images captured by the remote links 102 may be accessed by the client device 104. In addition, if a medical practitioner wants to watch a live video stream of a medical device with a remote link 102 integrated therein, the client device may display the video stream. The remote link architecture may include multiple client devices 104. As long as the client device can access the remote links 102, one client device 104 may access multiple remote links 102.

[0026] The WEB load balancer 108 controls the load on one or more WEB servers 114. The WEB servers 114 may include mechanisms for clients to view information, data, and video streams from one or more remote links 102. The WEB load balancer 108 may generate priorities for multiple client devices 104. The priorities may be based on preferences obtained from the client devices 104. In other aspects, the priorities are based on preferences obtained from the remote links 102. In other aspects, the priorities are based on preferences obtained from the WEB servers 114. The WEB socket server 122 may push messages to a group of client devices 104. When a client device 104 connects to the WEB server 114, the client device 104 registers with the WEB socket server 122 for messages of any of the one or more remote links 102. The WEB socket server 122 will receive messages applicable to the one or more remote links 102. This message along with the associated data is broadcast from those remote links 102 to all connected client devices 104 for updating.

[0027] The video load balancer 110 controls the load on one or more video servers 116. The video servers 116 may be receivers and senders of video streams from one or more remote links 102. The video load balancer 110 may generate priorities for the multiple client devices 104. The priorities may be based on preferences obtained from the client devices 104. In other aspects, the priorities are based on preferences obtained from the remote links 102. In another aspect, the priorities are based on preferences obtained from the video servers 116.

[0028] The RAM data type storage 118 may be a volatile storage that can be accessed quickly. The RAM data type storage 118 may include dynamic random access memory (DRAM), static random access memory (SRAM), or another type of fast volatile memory. Images captured by the remote link 102 may be stored in the RAM data type storage 118 before being sent to the client device 104. The RAM data type storage 118 may also store video streams captured by the remote link 102. The document data type storage 120 may be a non-volatile storage that can maintain data for a long period of time. The document data type storage 120 may be a hard disk, an optical disk, a solid state drive (SSD), or another type of non-volatile memory.

[0029] A process 200 for transferring images from a remote link 102 to a remote link router server 112 is shown in Figure 2. Process 200 begins with connecting the remote link 102 to the Internet at step 202. Step 202 may include initializing the remote link 102, as described below in process 500 of Figure 5.

[0030] The process 200 continues with sending a first signal from the remote link 102 to the RLR 150 indicating that the remote link 102 is connected to the Internet, as step 204. The first signal may be sent directly to the RLR load balancer 106. In another aspect, the first signal may be sent directly to the RLR server 112.

[0031] The process 200 continues at step 206 with the RLR 150 sending a command to the remote link 102 to begin capturing an image. For example, the remote link 102 captures an image from a medical device using an image capture unit 626, described below.

[0032] The process 200 continues at step 208 with transferring the image from the remote link 102 to the RLR 150. For example, an RLR load balancer manages the transfer of the image from the remote link 102 to the RLR server 112. Once the image is transferred to the RLR server 112, the process 200 proceeds to step 210.

[0033] The process 200 continues at step 210 with broadcasting a second signal from the RLR 150 indicating that the remote link 102 has captured an image. For example, the RLR 150 broadcasts the second signal so that the web server 114 is notified that the RLR 150 has an image captured by the remote link 102.

[0034] The process 200 continues at step 212 with receiving at the WEB server 114 a second signal broadcast from the remote link 102. For example, the WEB server 114 receives the signal broadcast from the RLR 150 such that the WEB server 114 is notified that the RLR 150 has an image captured by the remote link 102.

[0035] The process 200 ends with storing the image at the web server 114 at step 214. The image may be stored in the RAM data type storage 118. For example, the RLR 150 transfers the image to the web server 114, which then transfers the image to the RAM data type storage 118. In one aspect, the RLR 150 may transfer the image directly to the RAM data type storage 118.

[0036] A process 300 for transferring a video stream from a remote link 102 to a client device 104 is shown in Figure 3. The process 300 begins at step 302 with sending a request from the client device 104 to a WEB server 114 to view a video stream from the remote link 102. The request may be routed through a WEB load balancer 108 before being sent to the WEB server 114. In one aspect, the request may include information identifying the remote link 102 to be accessed.

[0037] The process 300 continues at step 304 with broadcasting a request from the WEB server 114. For example, the WEB server 114 notifies the RLRs 150 that the client device 104 has requested to view a video stream from the remote link 102 by broadcasting the request to all RLRs 150.

[0038] The process 300 continues at step 306 with receiving the request at the RLR 150. For example, the RLR server 112 receives the request from the web server 114. In one aspect, the RLR 150 receives the request after determining that the request identifies a remote link 102 communicatively coupled to the RLR 150.

[0039] The process 300 continues at step 308 with sending a command from the RLR 150 to the remote link 102 to transmit the video stream to the video server 116. For example, the RLR server 112 sends a signal to the remote link 102 via the RLR load balancer 106 to initiate the process of transmitting the video stream from the remote link 102 to the video server 116.

[0040] The process 300 continues at step 310 with sending the video stream from the remote link 102 to the video server 116. In one aspect, the remote link 102 sends the video stream to a video load balancer 110, which determines which video server 116 to send the video stream to. The video load balancer 110 may make the decision based on the load of the video server 116 and the priority of the remote link 102 and the client device 104.

[0041] The process 300 continues at step 312 with receiving the video stream at the video server 116. For example, once the video load balancer 110 determines which video servers 116 are available to receive the video stream, the video server 116 receives the video stream.

[0042] The process 300 concludes with transmitting the video stream from the video server 116 to the client device 104. For example, the video server 116 begins forwarding the video stream to the client device 104 via the video load balancer 110.

[0043] 4 shows a schematic diagram of a remote link architecture 400. The remote link architecture 400 includes a remote link 402, a client device 404, an RLR 450, a document data type storage 420, an HTTP service 430, and a cloud 460.

[0044] The remote link 402 is similar to the remote link 102 and may be incorporated into medical equipment monitoring a patient in a hospital, clinic, the patient's home, or another location. The remote link 402 may capture images and deliver video streams from the medical equipment display and transmit images and video to a remote link router 450. The remote link architecture 400 may include multiple remote links 402. The remote link 402 communicates with other parts of the remote link architecture 400 via the RLR 450. The RLR 450 is similar to the RLR 150 described above.

[0045] The client device 404 is similar to the client device 104 and may be a personal computer, tablet, or mobile device with an Internet connection. A medical practitioner using the client device 404 may want to obtain information from one or more remote links 402. Images captured by the remote links 402 may be accessed by the client device 404. In addition, if a medical practitioner wants to watch a live video stream of a medical device with an integrated remote link 402, the client device may display the video stream. The remote link architecture may include multiple client devices 404. One client device 404 may access multiple remote links 402, as long as the client device can access the remote links 402. The client device 404 may communicate with the RLR 450 via the cloud 460. The cloud 460 represents a network of Internet-based devices and connections, such as servers, storage, applications, etc.

[0046] The document data type storage 420 may be similar to the document data type storage 120 and may be a non-volatile storage capable of maintaining data for a long period of time. The document data type storage 420 may be a hard disk, an optical disk, a solid state drive (SSD), or another type of non-volatile memory. The document data type storage 420 may store Wi-Fi authentication information or other initial configuration information obtained from one or more client devices 404 or from the RLR 450. The document data type storage 420 may transmit the Wi-Fi authentication information or other initial configuration information to the RLR 450 or directly to one or more remote links 402.

[0047] The HTTP service 430 may be a framework that provides the ability for the RLR 450 to make HTTP requests. The RLR 450 may use the HTTP service 430 to obtain Wi-Fi authentication information or other initial configuration information and store that information in the document data type storage 420.

[0048] A process 500 for initializing the remote link 402 is shown in Figure 5. The process 500 begins with connecting the remote link 402 to an LTE network at step 502. In another aspect, the remote link 402 may connect to a 3G or 4G network.

[0049] The process 500 continues with the remote link 402 sending a first signal to the RLR 450 indicating that the remote link 402 is connected to the LTE network at step 504. For example, when the remote link 402 comes online, the remote link 402 sends a signal to the RLR 450 to notify the RLR 450 that it is ready to transmit or receive data. In one aspect, the RLR 450 is also connected to the LTE network.

[0050] The process 500 continues at step 506 with receiving the Wi-Fi authentication information at the RLR 450 from the client device 404. For example, a user enters the Wi-Fi authentication information into the client device 404, which then transmits the Wi-Fi authentication information to the RLR 450. In one aspect, the RLR 450 stores the Wi-Fi authentication information.

[0051] The process 500 then transmits the Wi-Fi authentication information from the RLR 450 to the remote link 402 at step 508. For example, the RLR 450 transmits the Wi-Fi authentication information to the remote link 402 using an LTE network.

[0052] Process 500 continues with connecting remote link 402 to a Wi-Fi network corresponding to the Wi-Fi credentials at step 510. For example, when remote link 402 receives the Wi-Fi credentials, remote link 402 searches for and connects to a Wi-Fi network identified by the Wi-Fi credentials.

[0053] The process 500 ends with the remote link 402 sending a second signal to the RLR 450 indicating that the remote link 402 is connected to the Wi-Fi network. For example, to confirm that the remote link 402 has successfully connected to the Wi-Fi network, the remote link 402 sends a signal to the RLR 450 indicating the successful connection using the Wi-Fi network. In another aspect, the remote link 402 sends a signal to the RLR 450 using the LTE network if the connection is faster than the Wi-Fi network. In one aspect, if the remote link 402 cannot connect to the Wi-Fi network, the remote link 402 sends a signal to the RLR 450 using the LTE network indicating that the connection was not successful.

[0054] 6 shows a schematic diagram of a remote link architecture 600. The remote link architecture 600 includes a medical device 624, a remote link 102, and a media server 116. The medical device 624 may include a sensor 626. The remote link 102 may include an image capture unit 628. The media server 116 may include an optical character recognition unit 630 and a motion data unit 632.

[0055] The medical device 624 may be a medical device monitoring a patient in a hospital, a clinic, the patient's home, or another location. The medical device 624 includes a sensor 626 that may be measuring and recording health signals from the patient. The sensor 626 may be a pressure sensor, a temperature sensor, a flow sensor, a voltage sensor, a current sensor, a light sensor, or an audio sensor.

[0056] The image capturer 628 may be an application that allows the remote link 102 to capture an image from the sensor 626. For example, the image capturer 628 captures an image of the display of the medical device 624. The image of the display of the medical device 624 may include data from the sensor 626 represented alphanumeric or graphically in a waveform plot. The image capturer 628 may convert analog data captured from the sensor 626 into digital data that may be used by the optical character recognition unit 630. For example, the image capturer 628 converts an analog signal from a video graphics array (VGA) connection from the sensor 626. As further described in connection with FIGS. 10-14, optical character recognition (OCR) may be used to convert images of text or shapes into digital data. In another aspect, other OCR equivalents and / or digital signal processing (DSP) may be used to extract data from the image.

[0057] The OCR unit 630 can be an application that electronically converts images of text or shapes into digital data. For example, the OCR unit 630 analyzes images captured by the image capture unit 628 on the remote link 102 to extract data from data embedded in the images. In some cases, the OCR unit 630 can extract data from waveforms.

[0058] In one aspect, the media server 116 may include a DSP unit 634. The DSP unit 634 may be an application that converts images into digital data. For example, the DSP unit 634 converts images captured by the image capture unit 628 on the remote link 102 into digital data. Once in digital form, the media server 116 may identify and / or filter motion and / or medical data embedded in the image. In another aspect, the DSP unit 634 may be used to extract data from a waveform contained in the image. For example, the OCR unit 630 extracts a period from the waveform portion of the image, and the DSP unit 634 uses the period and boundaries of the waveform to extract the motion and / or medical data. By using the period and boundaries of the waveform portion of the image, the DSP unit 634 associates pixels of the waveform portion with units of time. In some aspects, the OCR unit 630 is used to extract units of measurement from the waveform portion of the image, and the DSP unit 634 uses the period and units of measurement to extract the motion and / or medical data. For example, the OCR unit 630 may determine that the waveform portion of the image represents a positioning signal and / or motor current over a 10 second period, and the DSP unit 634 may associate each pixel in the waveform portion with a corresponding positioning signal and / or motor current and a unit of time equal to the period divided by the number of pixels in the waveform portion of the image.

[0059] The operational and / or medical data section 632 may be an application that databases and organizes data extracted from the OCR section 630 and / or DSP section 634. For example, the operational data section 632 identifies the type of data extracted by the OCR section 630 and / or DSP section 634 and categorizes the data into operational and / or medical conditions. The operational and / or medical conditions may include pressure, flow rate, pump speed, temperature, voltage, current, and biometric conditions.

[0060] The remote link architecture 600 may be implemented by processes 200, 300, and 500 to control the bandwidth, quality, and type of video streaming from the remote link device 102. The remote link architecture 600 may be extended to an unlimited number of remote link devices 102 and client devices 104. The OCR unit 630 and the operational data unit 632 may be included in other components of the remote link architecture 100, the remote link architecture 400, the remote link architecture 600, or the remote link architecture 700 (described below).

[0061] 7 is a schematic diagram of a remote link architecture 700. The remote link architecture 700 includes a remote link 102, a client device 104, an RLR 150, a media server 116, a web socket server 122, a web server 114, a cloud 460, a RAM data type storage 118, a document data type storage 120, and a message service 770.

[0062] The remote link 102 may be embedded in medical equipment monitoring a patient in a hospital, clinic, the patient's home, or another location. The remote link 102 may capture images and deliver video streams from the medical equipment display and transmit images and video to a remote link router 150. The remote link architecture 100 may include multiple remote links 102. The remote links 102 communicate with other parts of the remote link architecture 100 via the RLR 150.

[0063] The client device 104 may be a personal computer, tablet, or mobile device with an Internet connection. A medical practitioner using the client device 104 may want to obtain information from one or more remote links 102. Images captured by the remote links 102 may be accessed by the client device 104. In addition, if a medical practitioner wants to watch a live video stream of a medical device with a remote link 102 integrated therein, the client device may display the video stream. The remote link architecture may include multiple client devices 104. As long as the client device can access the remote links 102, one client device 104 may access multiple remote links 102.

[0064] The WEB server 114 may include a mechanism for clients to view information, data, and video streams from one or more remote links 102. The WEB socket server 122 may push messages to a group of client devices 104. When a client device 104 connects to the WEB server 114, it registers with the WEB socket server 122 for messages of any of the one or more remote links 102. The WEB socket server 122 will receive messages applicable to one or more remote links 102. The messages, along with associated data, are broadcast from those remote links 102 to all connected client devices 104 for updates. The message service 770 may manage the transfer of messages between different components of the remote link architecture 700 via the cloud 460. The cloud 460 represents a network of Internet-based devices and connections, such as servers, storage, applications, etc.

[0065] The media server 116 may be a receiver and sender of video streams from one or more remote links 102. The media server 116 may be similar to the video server 116 described above. The media server 116 may also be a receiver and sender of images captured from one or more remote links 102.

[0066] The RAM data type storage 118 may be a volatile storage that can be accessed quickly. The RAM data type storage 118 may include dynamic random access memory (DRAM), static random access memory (SRAM), or another type of fast volatile memory. Images captured by the remote link 102 may be stored in the RAM data type storage 118 before being sent to the client device 104. The RAM data type storage 118 may also store video streams captured by the remote link 102. The document data type storage 120 may be a non-volatile storage that can maintain data for a long period of time. The document data type storage 120 may be a hard disk, an optical disk, a solid state drive (SSD), or another type of non-volatile memory.

[0067] FIG. 8 illustrates an example of a medical device, such as an intravascular blood pump 800, according to certain embodiments. The pump 800 includes a pump handle 810, a pump head 830, a catheter 820 that connects the pump handle 810 to the pump head 830, and a connection hub 860. The catheter 820 is tubular and has a substantially uniform outer diameter 850. The catheter 820 allows the pump head 830 and the pump handle 810 to be in electromechanical communication. The pump handle 810 is in communication with control circuitry that allows control of the pump head 830. The pump head 830 includes electromechanical components that allow the device to perform various tasks within the patient's body, such as pumping blood from a location within the body. The pump head 830 has a diameter 840 that is larger than the diameter 850 of the catheter 820. One example of such a percutaneous pump is the Impella 2.5® system (Abiomed, Inc., Danvers, Massachusetts), which includes a pump and an Automatic Impella Controller (AIC).

[0068] 9 illustrates an example medical device controller 900, such as an AIC, configured in accordance with one or more aspects of the disclosure. The medical device controller 900 provides an interface for monitoring and controlling the functions of the pump 800. The medical device controller 900 may include a display screen 902 that may display images from a video stream, the images representing data associated with a medical device, such as the intravascular blood pump 800, over a period of time. In certain implementations, the display screen 902 displays real-time operational and / or medical data associated with the pump 800.

[0069] FIG. 10 illustrates an exemplary image 1000, for example, displayed on a display screen 902, configured according to one or more aspects of the disclosure. In some configurations, the image 1000 may be captured by an intermediate device or data module, such as the remote link 102, over a network and transmitted to another device, such as the media server 116. The image 1000 may include a waveform 1002. The waveform 1002 represents medical and / or operational data corresponding to the operation of the pump 800. Examples of medical data represented by the waveform 1002 include positioning signals and motor currents. The waveform 1002, such as a motor current waveform, may provide a history, representation, and / or diagram of the motor current over a period of time (e.g., 10 seconds). In this manner, the image 1000 includes motor current data (and other data) associated with the pump 800 over a 10 second period. Thus, in one embodiment, the data module 102 constantly monitors the video stream output from the appliance controller 900, but only periodically captures images such as the image 1000. The data module 102 then transmits the image 1000 to another device, such as the server 116, which converts the represented waveform 1002 into medical and / or operational data, for example using OCR. For example, if the waveform 1002 represents 10 seconds of medical data, the data module 102 may capture successive images 1000 every 10 seconds (at 10 second intervals) so that there are no gaps in the data provided to the server 116. Processes 1300 and 1400, discussed below in connection with Figures 13 and 14, respectively, describe exemplary methods of extracting data from an image and determining the validity of the extracted data.

[0070] In one aspect, the server 116 masks certain portions of the image 1000 before extracting data using the OCR unit 630 or equivalent. FIG. 11 illustrates an example image 1100, configured in accordance with one or more aspects of the present disclosure. The image 1100 is a masked version of the image 1000 in which certain portions of the image 1000 have been removed. For example, all portions of the image 1000 are removed except for the alarm and serial number portion 1102, the performance level portion 1104, and the flow level portion 1106. After generating the image 1100, the server 116 performs image processing to clarify and expand the alarm and serial number portion 1102, the performance level portion 1104, and the flow rate portion 1106.

[0071] FIG. 12 illustrates an example image 1200, configured in accordance with one or more aspects of the present disclosure. Image 1200 is a processed version of image 1100 to facilitate extraction of data using OCR unit 630. In one aspect, alarm and serial number portion 1102 may be processed into serial number portion 1202 and alarm portion 1204. Serial number portion 1202 includes a specific number of digits that identifies the medical device 624 currently being monitored and may be expanded to facilitate OCR. For example, serial number portion 1202 includes six digits. Alarm portion 1204 may indicate a type of alarm that may be occurring on medical device 624. For example, alarm portion 1204 includes color pixels that indicate a severity of an alarm that may be occurring on medical device 624. Example colors for alarm portion 1204 include red, yellow, and green. In some aspects, performance level portion 1206 indicates a performance level of pump 800 and includes three letters. Examples of characters in the performance level portion 1206 may include "OFF", "P-0", "P-1", "P-2", "P-3", "P-4", "P-5", "P-6", "P-7", "P-8", and "P-9". The performance level portion 1206 may be an expanded version of the performance level portion 1104. In another aspect, the flow rate portion 1106 may be processed into a current flow rate portion 1208, a maximum flow rate portion 1210, and a minimum flow rate portion 1212. The current flow rate portion 1208 indicates the current flow rate of the pump 800 in units of liters per minute. Correspondingly, the maximum flow rate portion 1210 and the minimum flow rate portion 1212 may each indicate a range of flow rates of the pump 800 and may be expanded to facilitate OCR. The current flow rate portion 1208, the maximum flow rate portion 1210, and the minimum flow rate portion 1212 each include three characters ranging from "0.0" to "9.9".

[0072] A process 1300 for extracting data from an image is shown in Figure 13. The process 1300 begins with receiving a first image representing data from a medical device 624 at step 1302. For example, the remote link 102 captures an image 1000 using the image capturer 628, and the server 116 receives the image 1000 from the remote link 102.

[0073] The process 1300 continues with masking the first portion of the first image at step 1304. For example, the server 116 uses an image mask to mask portions of the image 1000 that are not sent to the OCR unit 630 for data extraction. Masking selected portions of the image can improve the efficiency of image processing because only the unmasked selected portions of the image are sent to the OCR unit 630 or the DSP unit 634. Masking selected portions of the image can result in less data being transmitted between the server 116 and the OCR unit 630 and the DSP unit 634, and the OCR unit 630 and the DSP unit 634 require less processing to extract data from the image. In one aspect, the server 116 can generate the image 1100 by removing certain portions of the image 1000 from the image 1000 using an image mask. For example, the server 116 generates the image 1100 by using an image mask to strip out all portions of the image 1000 except the alarm and serial number portion 1102, the performance level portion 1104, and the flow level portion 1106. In another aspect, the server 116 may select a different mask that corresponds to a feature of the image 1000. For example, the server 116 may select a different mask based on the size of the image 1000 or the GUI version that corresponds to the image 1000. For example, the server 116 may select a mask based on the software version of the remote link 102. In some aspects, the server 116 may select a mask based on the type of display screen 902 being used. For example, if the image displayed on the display screen 902 is not an appropriate image for the first mask selected by the server 116, the server 116 may determine that the first mask used is not an appropriate mask for the image 1000 and select a different mask based on the image currently displayed on the display screen 902. In one aspect, the server 116 may wait until an appropriate image is displayed on the display screen 902 before masking a portion of the image 1000. In another aspect, the server 116 may select the mask based on the amount of data to be extracted from the image 1000 .

[0074] Process 1300 continues with generating a second image using the remaining portions of the first image at step 1306. For example, server 116 performs image processing to generate image 1200 by clarifying and enlarging alarm and serial number portion 1102, performance level portion 1104, and flow rate portion 1106. In one aspect, server 116 may generate serial number portion 1202 and alarm portion 1204 from serial number portion 1102, performance level portion 1206 from performance level portion 1104, and current flow rate portion 1208, maximum flow rate portion 1210, and minimum flow rate portion 1212 from flow rate portion 1106.

[0075] The process 1300 ends with extracting data from the second image using optical character recognition at step 1308. For example, the serial number of the medical device 624, the type of alarm currently being indicated, the performance level of the pump 800, and the flow rate are extracted from the image 1200 using the OCR unit 630. In one aspect, the OCR unit 630 may select pixels from the second image to determine the alarm severity from the alarm portion 1204. For example, the OCR unit 630 may determine the color of the pixel and determine the alarm severity based on the color of the pixel. In some aspects, the OCR unit 630 may select two different pixels from the second image to determine the alarm severity from the alarm portion 1204. For example, the storage 120 stores a database of alarm types, alarm severities, and corresponding alarm colors. The server 116 may access the database stored in the storage 120 and determine the alarm type and severity associated with the color of one or more pixels from the alarm portion 1204. In another aspect, the OCR unit 630 may select a first pixel at a first time and a second pixel at a second time. For example, if the image 1000 is defective when received by the server 116, the server 116 cannot determine the color of the pixel from the second image at the first time. The server 116 receives another image 1000 to determine the color of another pixel from the second image at the second time. In another aspect, the server 116 determines the alarm severity to be the same as the severity of the previous alarm if the server 116 cannot determine the color of the pixel from the two pixels. In another aspect, the process 1400 described below may be used to validate the extracted data from the second image.

[0076] A process 1400 for determining the validity of data from an image is shown in Figure 14. The process 1400 begins with extracting first data from a first portion of the image using optical character recognition at step 1402. For example, the serial number of the medical device 624 is extracted from the serial number portion 1202. In one aspect, the extraction of the first data from the first portion of the image may be performed using the process 1300 described above.

[0077] The process 1400 continues with comparing the first data to reference data at step 1404. In one aspect, the reference data may include a specific number of letters and / or numbers representing a standard format in which the first data may be represented. For example, the extracted serial number of the medical device 624 is compared to possible serial numbers stored in the document data type storage 120. Further examples of comparing data to reference data are described in U.S. Patent No. 9,002,083, entitled "System, Method, and Software for Optical Device Recognition Association," the entire contents of which are incorporated herein by reference.

[0078] The process 1400 continues at step 1406 by determining the validity of the first data based on the comparison. For example, if the extracted serial number of the medical device 624 does not match a standard format for serial numbers, for example, consisting of a certain number of letters and / or numbers, the extracted serial number is not valid. In one aspect, if the extracted serial number does not include six digits and the standard format for serial numbers is six digits, the extracted serial number is not valid. In another aspect, step 1406 is repeated a certain number of times before making a final determination. For example, if three attempts are required to validate the first data, the first data is determined to be valid if the comparison results match three times. If one of the comparison results does not match during the three attempts, the first data is determined to be invalid.

[0079] In response to determining that the first data is valid based on the comparison, the process 1400 proceeds to step 1408. At step 1408, the process 1400 continues by extracting second data from a second portion of the second image using optical character recognition. For example, a performance level of the pump 800 is extracted from the performance level portion 1206. As described in connection with FIG. 12, example characters in the performance level portion 1206 may include "OFF", "P-0", "P-1", "P-2", "P-3", "P-4", "P-5", "P-6", "P-7", "P-8", and "P-9". In one aspect, the process 1400 may proceed to step 1402 until all data from each portion of the image 1200 has been extracted.

[0080] In response to determining that the first data is not valid based on the comparison, the process 1400 proceeds to step 1410. At step 1410, the process 1400 continues by broadcasting a signal indicating that the first data is not valid. For example, the server 116 notifies the remote link 102 that the image 1000 generated invalid first data.

[0081] The process 1400 ends with receiving a third image representative of the data from the medical device at step 1412. For example, the remote link 102 may capture another image similar to 1000 using the image capturer 628, and the server 116 may receive the similar image from the remote link 102. In one aspect, the process 1400 may continue at step 1402 until all data from each portion of the image 1200 has been extracted.

[0082] Although a percutaneous heart pump is described herein, it will be understood that any other medical device may be used in conjunction with the present disclosure. Additionally, while Figures 8 and 9 show a media device configuration in which the controller 900 is separate from the pump 800, one of ordinary skill in the art will readily appreciate that the medical device may be configured such that the controller and pump (or other elements) are integrated within the same housing.

[0083] Other objects, advantages and aspects of the various aspects of the present invention will be apparent to those skilled in the art of the present invention and are within the scope of this description and the accompanying drawings. For example, and without limitation, structural or functional elements may be rearranged consistent with the present invention. Similarly, the principles according to the present invention may be applied to other examples that are within the scope of the present invention, even if not specifically described in detail herein.

Claims

1. A method performed by a server, comprising: receiving, by the server, a first image representative of data from a medical device; masking, by the server, a first plurality of portions of the first image, the first image including the first plurality of portions and a second plurality of portions; modifying, by the server, one or more of the second plurality of portions by expanding at least one of the second plurality of portions or separating at least one of the second plurality of portions into two or more portions; generating, by the server, a second image including the one or more modified portions; extracting data from the second image by an optical character recognition (OCR) unit of the server; A method comprising:

2. The method of claim 1 , wherein the first image is captured from a video stream from the medical device.

3. Extracting the data from the second image extracting, by the OCR unit, first data from a first portion of the second image; determining, by the server, validity of the first data by comparing the first data with reference data; extracting, by the OCR unit, second data from a second portion of the second image in response to determining that the first data is valid; and The method of claim 1 , comprising:

4. in response to determining that the first data is not valid, broadcasting, by the server, a signal indicating that the first data is not valid; receiving, by the server, a third image representative of data from the medical device; The method of claim 3, further comprising:

5. Masking the first plurality of portions of the first image includes: selecting, by the server, an image mask based on a size of the first image; masking, by the server, the first plurality of portions using the image mask; The method of claim 1 , comprising:

6. The method of claim 1, further comprising extracting waveform data from a waveform in the second image by a digital signal processing (DSP) unit of the server.

7. selecting, by the server, a first pixel from the second image that includes a color; determining, by the server, the color of the first pixel, the color corresponding to an alarm severity; determining, by the server, the alarm severity based on the color of the first pixel; The method of claim 1 , further comprising: selecting, by the server, a second pixel from the second image; determining, by the server, the alarm severity based on the first pixel and the second pixel; The method of claim 7, further comprising:

9. The method of claim 1, wherein modifying the one or more portions of the second plurality of portions includes enlarging at least one of the second plurality of portions.

10. The method of claim 9, wherein the medical device is an intravascular blood pump and the at least one of the second plurality of portions includes a serial number portion, an alarm portion, a performance level portion, or a flow rate portion.

11. The method of claim 1, wherein modifying the one or more portions of the second plurality of portions includes separating at least one of the second plurality of portions into two or more portions.

12. The method of claim 11, wherein the medical device is an intravascular blood pump and the at least one of the second plurality of portions includes an alarm and a serial number portion.

13. The method of claim 12, wherein the alarm and serial number portion is separated into a serial number portion and an alarm portion.

14. The method of claim 11, wherein the medical device is an intravascular blood pump and the at least one of the second plurality of portions includes a flow portion.

15. The method of claim 14, wherein the flow rate portion is separated into a current flow rate portion, a maximum flow rate portion, and a minimum flow rate portion.

16. The method of claim 1, wherein a position of at least one of the modified portions in the second image is different from a position of a corresponding one of the second plurality of portions in the first image.

17. A non-transitory computer-readable storage medium having stored thereon instructions, the instructions, when executed by one or more processors, causing the one or more processors to: Receiving a first image representative of data from a medical device; masking a first plurality of portions of the first image, the first image including the first plurality of portions and a second plurality of portions; modifying one or more of the second plurality of portions by enlarging at least one of the second plurality of portions or separating at least one of the second plurality of portions into two or more portions; generating a second image including the one or more modified portions; extracting data from the second image by an optical character recognition (OCR) unit; and A non-transitory computer-readable storage medium that causes

Images