Intelligent Image Segmentation Prior to Optical Character Recognition (OCR)

Abstract

To provide: a system capable of intelligently segmenting images of screens displayed by a medical device controller before subjecting the images to optical character recognition (OCR) processing; a method; and a computer readable medium.SOLUTION: A medical device monitoring method includes the steps of: extracting information from screen images from medical device controllers, with a single OCR process invocation per screen image, despite critical information appearing in different screen locations, depending on which medical device controller's screen image is processed; creating a mosaic image one copy at a time from each screen position with different important screen information; subjecting the mosaic image to OCR processing to produce text results; and selectively extracting text from the OCR text results according to the contents of a selector field on the screen image such as a software version number or a heart pump model identifier.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Patent Application No. 16 / 360,270, filed March 21, 2019, entitled "Intelligent Image Segmentation Before Optical Character Recognition (OCR)," the entire contents of which are incorporated herein by reference for all purposes.

[0002] Technical Field The present invention relates to remote monitoring of medical devices, and more particularly to intelligently segmenting images of a screen displayed by a medical device controller prior to optical character recognition (OCR) processing of the image. [Background technology]

[0003] Related technologies Many medical devices, such as some intravascular blood pumps, e.g., the Impella® 2.5 heart pump available from Abiomed, Inc. of Danvers, Massachusetts, are connected to external medical device controllers that collect and display operational data about the medical device, such as cardiac signal levels, battery temperature, blood flow, and tubing integrity. An exemplary medical device controller is available from Abiomed, Inc. under the trade name Automated Impella Controller®. These medical device controllers issue alarms when operational data values ​​exceed predetermined values ​​or ranges, for example, when a leak or loss of suction is detected. These medical device controllers include a video display screen as a human interface, on which the operational data and / or alarms are displayed.

[0004] To facilitate remote monitoring by medical personnel to ensure effectiveness and patient safety, some such medical device controllers may be coupled, often via a computer network including a wireless segment, to a central server that can be accessed by a monitoring station that may display real-time operating data and / or alarms on a display screen for viewing by medical personnel.

[0005] Servers request and receive images of content displayed on the screen of a medical device controller. Some servers use optical character recognition (OCR) technology to parse the images and extract text information, such as heart pump serial numbers, blood flow rates, and warning message text. To reduce OCR errors, some servers mask portions of the screen image that contain irrelevant text or graphics that may confuse the OCR engine.

[0006] However, different medical device controllers may display the same type of information, such as blood flow rate, in different locations on their screens, depending, for example, on which version of software the medical device controller is running or which type of heart pump is connected to the medical device controller. This ambiguity in screen location makes it difficult or impossible to effectively mask the appropriate portion of the screen image before OCR. Therefore, the server may use multiple masks, perform an OCR process with each mask, and then select one OCR result. However, OCR processing is expensive and time-consuming. Providers would prefer to have the server perform only one OCR process per screen image. Unfortunately, the server has no foresight and therefore cannot determine in advance which mask to use. Summary of the Invention

[0007] Overview of Aspects One aspect of the present invention provides a medical device monitoring system. The system includes a server. The server is configured to automatically request and receive "incoming images." The server is configured to request and receive the incoming images via a computer network. The server is configured to request and receive the incoming images from each medical device control device of a plurality of medical device control devices. Each incoming image includes content displayed on a screen of the medical device control device. The plurality of medical device control devices includes a plurality of first medical device control devices and a plurality of second medical device control devices. For example, the first medical device control device may run a particular version of software, and the second medical device control device may run a different version of software. In another example, different types of medical devices are connected to the first medical device control device and the second medical device control device, respectively.

[0008] Each first medical device controller is configured to display a first type of information at a first screen coordinate. Each second medical device controller is configured to display the first type of information at a second screen coordinate that is different from the first screen coordinate. In other words, the first medical device controller and the second medical device controller display certain information ("first information"), such as minimum blood flow, at different locations on their respective screens. This difference in screen location may be due to different software versions or different types of medical devices connected to the first and second medical device controllers.

[0009] For each received image, the server is configured to automatically generate a mosaic image. To generate the mosaic image, the server is configured to copy a first source region of the received image to a predefined destination region at a third coordinate in the mosaic image. The first source region encompasses a first screen coordinate of the received image. The server is also configured to copy a second source region of the received image to a predefined destination region at a fourth coordinate in the mosaic image. The fourth coordinate is different from the third coordinate. The second source region encompasses a second screen coordinate of the received image. The server is further configured to copy multiple additional source regions of the received image to respective additional predefined destination regions in the mosaic image.

[0010] After the mosaic image is generated, the server is configured to automatically subject the mosaic image to optical character recognition in a single service call to generate text results.

[0011] The server is also configured to automatically extract a selector text result from the text results. The selector text result corresponds to at least one of the additional predefined regions of interest in the mosaic image. The server is configured to compare the selector text result with a first predefined text. If the selector text result is found to match the first predefined text, the server is configured to extract a selected text result from the text results, such that the selected text result corresponds to a third coordinate in the mosaic image. Otherwise, the server is configured to extract a selected text result from the text results, such that the selected text result corresponds to a fourth coordinate in the mosaic image. The server is configured to provide the selected text result.

[0012] Thus, the medical device control device may display the same information in different screen locations, but the server can select the OCR result from the appropriate one of the screen locations based on the content of another field on the screen, without having to perform two OCR processes.

[0013] In any embodiment, the server may be configured to automatically compare the selector text result with a second predefined text if the selector text result is found to not match the first predefined text, the second predefined text being different from the first predefined text. If the selector text result is found to match the second predefined text, the server may be configured to automatically extract the selected text result from the text result, such that the selected text result corresponds to a fourth coordinate in the mosaic image.

[0014] In any embodiment, at least one of the additional defined regions of interest in the mosaic image may correspond to an additional source region containing a version number of software executed by the medical device controller.

[0015] In any embodiment, at least one of the additional defined regions of interest in the mosaic image may correspond to an additional source region that includes an identifier of a medical device coupled to the medical device controller.

[0016] In any embodiment, the medical device identifier may include a serial number.

[0017] In any embodiment, the medical device identifier may include a type code.

[0018] In any embodiment, at least one of the additional defined regions of interest in the mosaic image may correspond to an additional source region that includes an identifier of an optional feature installed on the medical device controller.

[0019] In any embodiment, the defined destination region at the third coordinate in the mosaic image may be larger than the first source region. To copy the first source region into the defined destination region at the third coordinate, the server may be configured to expand the first source region to fill the defined destination region at the third coordinate.

[0020] In any embodiment, to copy the first source region to the defined destination region at the third coordinate, the server may be configured to convert each colored pixel of the first source region to a white pixel in the defined destination region at the third coordinate.

[0021] Another aspect of the present invention provides a method for monitoring a medical device. The method includes requesting and receiving received images. The requested images are requested and received via a computer network. The received images are requested and received by each medical device controller of a plurality of medical device controllers. Each received image includes content displayed on a screen of the medical device controller. The plurality of medical device controllers includes a plurality of first medical device controllers and a plurality of second medical device controllers. Each first medical device controller is configured to display a first type of information at a first screen coordinate, and each second medical device controller is configured to display the first type of information at a second screen coordinate different from the first screen coordinate.

[0022] The method includes generating a mosaic image for each received image. The mosaic image generating step includes copying a first source region encompassing a first screen coordinate of the received image to a predefined destination region at a third coordinate in the mosaic image. A second source region encompassing a second screen coordinate of the received image is copied to a predefined destination region at a fourth coordinate in the mosaic image, the fourth coordinate being different from the third coordinate. Multiple additional source regions of the received image are copied to respective additional predefined destination regions in the mosaic image.

[0023] After generating the mosaic image, the mosaic image is optically character recognized in a single service call to generate a text result. A selector text result is extracted from the text result. The selector text result corresponds to at least one of the additional predefined regions of interest in the mosaic image. The selector text result is compared to the first predefined text.

[0024] If the selector text result is found to match the first predefined text, the selected text result is extracted from the text result, such that the selected text result corresponds to a third coordinate in the mosaic image. Otherwise, the selected text result is extracted from the text result, such that the selected text result corresponds to a fourth coordinate in the mosaic image. The selected text result is then provided.

[0025] In any embodiment, if not, extracting the selected text result may include comparing the selector text result to a second predefined text. The second predefined text may be different from the first predefined text. If the selector text result is found to match the second predefined text, the selected text result is extracted from the text result such that the selected text result corresponds to a fourth coordinate in the mosaic image.

[0026] In any embodiment, at least one of the additional defined regions of interest in the mosaic image may correspond to an additional source region containing a version number of software executed by the medical device controller.

[0027] In any embodiment, at least one of the additional defined regions of interest in the mosaic image may correspond to an additional source region that includes an identifier of a medical device coupled to the medical device controller.

[0028] In any embodiment, the medical device identifier may include a serial number.

[0029] In any embodiment, the medical device identifier may include a type code.

[0030] In any embodiment, at least one of the additional defined regions of interest in the mosaic image may correspond to an additional source region that includes an identifier of an optional feature installed on the medical device controller.

[0031] In any embodiment, the defined region of interest at the third coordinate in the mosaic image may be larger than the first source region, and copying the first source region to the defined region of interest at the third coordinate may include expanding the first source region to fill the defined region of interest at the third coordinate.

[0032] In any embodiment, copying the first source region to the defined destination region at the third coordinates may include converting each colored pixel of the first source region to a white pixel in the defined destination region at the third coordinates.

[0033] Yet another aspect of the present invention provides a non-transitory computer-readable medium encoded with instructions. When executed by a processor, the instructions establish a process for performing a computer-implemented method for monitoring medical devices. The process includes a process configured to request and receive, via a computer network, from each medical device controller of a plurality of medical device controllers, a received image of content displayed on a screen of the medical device controller. The plurality of medical device controllers includes a plurality of first medical device controllers, each configured to display a first type of information at a first screen coordinate, and a plurality of second medical device controllers, each configured to display the first type of information at a second screen coordinate different from the first screen coordinate.

[0034] The process also includes a process configured to generate a mosaic image for each received image. The process is configured to copy a first source region encompassing a first screen coordinate of the received image to a predefined destination region at a third coordinate in the mosaic image. The process is also configured to copy a second source region encompassing a second screen coordinate of the received image to a predefined destination region at a fourth coordinate in the mosaic image, the fourth coordinate being different from the third coordinate. The process is further configured to copy multiple additional source regions of the received image to respective additional predefined destination regions in the mosaic image.

[0035] The process further includes a process configured to, after generating the mosaic image, subject the mosaic image to optical character recognition in a single service call to generate a text result.

[0036] The process includes a process configured to extract a selector text result from the text result, the selector text result corresponding to at least one of the additional predefined regions of interest within the mosaic image, and a process configured to compare the selector text result with the first predefined text.

[0037] The process includes a process configured to, if the selector text result is found to match the first predefined text, extract the selected text result from the text result such that the selected text result corresponds to a third coordinate in the mosaic image, and otherwise, extract the selected text result from the text result such that the selected text result corresponds to a fourth coordinate in the mosaic image.

[0038] The processes also include a process configured to provide selected text results. [Brief explanation of the drawings]

[0039] The present invention will be more fully understood from the following detailed description of specific embodiments taken in conjunction with the drawings.

[0040] [Figure 1] 1 is a perspective view of an exemplary conventional medical device control device and an exemplary conventional medical device, in this example a heart pump, coupled to the medical device control device, according to the prior art. [Figure 2] 2A-2C illustrate exemplary virtual display screen contents that may be displayed on the screen of the medical device control device of FIG. 1 according to the prior art. [Figure 3] 3A-3C illustrate schematically steps for processing a screen display image such as the virtual display screen content of FIG. 2 according to one embodiment of the present invention. [Figure 4]10A and 10B show another exemplary virtual display screen content that may be displayed on the screen of another medical device controller, such as a medical device controller running a different version of software than the medical device controller that generated the screen content shown in FIG. 2, according to the prior art. [Figure 5] FIG. 5 illustrates a schematic diagram of a process for processing a screen display image such as the virtual display screen content of FIG. 2 or the virtual display screen content of FIG. 4, in accordance with one embodiment of the present invention, regardless of the differences between the virtual display screen content of FIG. 2 and the virtual display screen content of FIG. 4. [Figure 6] 6 is a schematic block diagram of the main components of a medical device monitoring system for collecting, storing, and retrieving operational data relating to multiple medical device control devices from multiple medical device control devices, including a server that performs the processing steps shown in FIG. 5, in accordance with one embodiment of the present invention. [Figure 7] FIG. 7 is a schematic block diagram of the server of FIG. 6 according to an embodiment of the present invention. [Figure 8] 8 is a flow chart that schematically illustrates a method for monitoring medical devices, such as the method performed by the server of FIGS. 6 and 7, in accordance with an embodiment of the present invention. [Figure 9] 9 is a flowchart that schematically illustrates operational details of the flowchart of FIG. 8 (selecting one text result based on additional text results), in accordance with an embodiment of the present invention. [Figure 10] 8 is a flowchart that schematically illustrates operations performed by any color pixel whitener of the server of FIGS. 6 and 7, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0041] Detailed Description of Specific Embodiments Aspects of the present invention provide a medical device monitoring system and method that extracts information from screen images from medical device control devices with a single OCR process invocation per screen image, even though important information appears in different screen locations depending on which medical device control device screen image is processed.

[0042] FIG. 1 is a perspective view of an exemplary conventional medical device controller 100 and an exemplary conventional medical device 102, in this example, a heart pump, coupled to the medical device controller 100. In the example shown in FIG. 1, the medical device controller 100 is an Automated Impella Controller® manufactured by Abiomed, Inc. of Danvers, Massachusetts, and the heart pump 102 is an Impella® 2.5 heart pump, also available from Abiomed, Inc., although any suitable medical device controller may be used. In some cases, a medical device and its associated medical device controller are combined. Such a combination is referred to herein simply as a medical device controller.

[0043] The medical device controller 100 includes a display screen 104 on which it displays operational data related to the medical device, such as cardiac signal levels, battery temperature, blood flow, and tubing integrity. As discussed in more detail herein, the medical device controller 100 may be connected to a computer network, thereby transmitting images of the content displayed on the screen 104 to a remote server (not shown).

[0044] Figure 2 shows exemplary virtual display screen content 200 that may be displayed on the screen 104 of the medical device controller 100 of Figure 1. For example, the display screen content may include a heart pump type ("Impella 5.0") 202, a heart pump serial number ("120703") 204, one or more warning / error / information messages 206, a placement signal 207, a current heart pump speed (performance) setting ("P-0") 208, a heart pump motor current value 210, a current blood flow rate 212, and minimum and maximum blood flow rates 213. The display screen content 200 is typically pixelated.

[0045] Figure 3 illustrates, in a simplified manner, a process for processing a screen display image 300, such as virtual display screen content 200 of Figure 2. While Figure 3 is a schematic diagram, portions of Figure 3, such as screen image 300, illustrate the approximate locations and sizes of certain items, such as heart pump type source region 306 and graphs 318 and 320, that are consistent with the exemplary virtual display screen content 200 shown in Figure 2.

[0046] As noted above, to reduce OCR errors, some servers mask portions of the screen image 200, 300 that contain extraneous text or graphics that may confuse the OCR engine 302. For example, the virtual display screen content 200 (FIG. 2) includes graphs 214 and 216 and legends such as "Impella Flow" 218, which, if included in the input to the OCR engine 302, may cause or contribute to OCR errors.

[0047] To avoid these errors, mosaic image 304 is generated from only a predetermined portion of screen display image 300. Mosaic image 304 may, but need not, have the same dimensions in pixels as screen display image 300. In this example, three source regions 306, 308, and 310 of screen display image 300 are copied into respective defined destination regions 312, 314, and 316 of mosaic image 304. However, any number of source regions of screen display image 300 may be copied into mosaic image 304.

[0048] Each source region 306-310 may be identified by corresponding screen coordinates. As used herein, the term screen coordinates, or simply coordinates, refers to information that identifies the location and size of a region of an image, i.e., the overall space in which information is displayed or placed within the image. A region may be rectangular, but need not be. The coordinates may include, for example, the x and y pixel location of the region's lower-left corner, as well as the region's height and width in pixels. Other suitable units of measure and other suitable representations may be used, so long as they identify the location and extent of the region. Thus, fields of different widths have different coordinates even if both fields start at the same x and y location.

[0049] The destination regions 312-316 may, but need not, have the same coordinates as their respective corresponding source regions 306-310. That is, the destination regions 312-316 may, but need not, have the same starting position and / or size as their respective corresponding source regions 306-310. In some embodiments, one or more of the destination regions 312-316 is larger than the corresponding source region 306-310.

[0050] In this case, the server magnifies the source regions 312-316 so that the magnified results fill the corresponding destination regions 312-316. Exemplary magnification factors include, but are not limited to, 1.1x, 1.5x, 2x, and 3x magnification factors. Such magnification factors may reduce the likelihood of OCR errors.

[0051] In some embodiments, the server changes the color of pixels as they are copied from source regions 306-310 to destination regions 312-316. For example, the server may convert colored pixels to white pixels to increase contrast and therefore reduce OCR errors. For example, the server may convert to white pixels whose total luminance, determined by the sum of the pixel's red, blue, and green values, exceeds a threshold, and the server may convert all remaining pixels to black.

[0052] Irrelevant portions of screen display image 300, such as graphs 318 and 320 (corresponding to graphs 214 and 216 in FIG. 2), are not copied into mosaic image 304. The screen coordinates of source regions 306-310 should be selected to avoid text and / or graphics that, if included, could confuse OCR engine 302 or cause OCR engine 302 to include spurious text in its output. The remainder of mosaic image 304 is left blank.

[0053] Once the mosaic image 304 is generated, it is passed to an OCR engine 302 for conversion to text. The server may include the OCR engine 304, or the server may invoke an external OCR engine, such as the OCR engine of the Google Cloud Vision API provided by Google (Alphabet, Inc., Mountain View, California). The OCR engine 302 returns text results 321, represented by Text1 322, Text2 324, and Text3 326, which correspond to the target regions 312-316, respectively. The server extracts Text1 322, Text2 324, and Text3 326 from the text results 321.

[0054] Optionally, text result 321, or portions thereof, e.g., Text1, Text2, and / or Text3 in 322-326, may be validated using predefined rules, such as "Pump serial number must be a 5-7 digit number." Such validation may prevent some OCR errors that corrupt subsequent displays, etc. Optionally, text result 321, or portions thereof, 322-326, may also be cleaned up according to predefined procedures, such as "Remove leading and trailing spaces from error messages." These predefined rules and procedures may be stored in a configuration file, which is described in more detail herein.

[0055] As noted above, different medical device controllers 100 (FIG. 1) may display the same type of information, e.g., blood flow rate, in different locations on their respective screens 104 depending, for example, on which version of software the medical device controller is running or which type of heart pump 102 is connected to the medical device controller 100. FIG. 4 shows another exemplary virtual display screen content 400 that may be displayed on the screen 104 of another medical device controller 100, e.g., a medical device controller 100 running a different version of software than the medical device controller that generated screen content 200 (FIG. 2) or connected to a different type of heart pump 102, such as an Impella Routine CP heart pump manufactured by Abiomed, Inc.

[0056] In exemplary virtual display screen content 400, it can be seen that heart pump type (“Impella CP”) 402 differs from the corresponding heart pump type 202 shown in screen content 200 ( FIG. 2 ), indicating that a different type of heart pump is connected to the medical device controller that generated display screen content 400. The heart pump serial number is shown at 404. As can be seen by comparing FIG. 4 with FIG. 2 , some of the information in display content 400 is displayed in different locations than the corresponding information in display content 200. For example, placement signal 407, motor current 410, and current, minimum, and maximum blood flow rates 412 and 413 are each at screen coordinates (position and / or size) from the same information 207, 210, 212, and 213, respectively, in screen content 200 ( FIG. 2 ).

[0057] Clearly, this difference in screen coordinates creates a problem for a single server that receives screen images from both medical device controllers and attempts to mask the relevant portions of the screen in the manner discussed with respect to Figure 3. Simply defining a source region large enough to encompass both versions of the display would fail because such a large source region, e.g., source region 420, would encompass irrelevant text ("(mmHg)") in display content 400.

[0058] Aspects of the present invention solve this problem, as shown generally in Figure 5. While Figure 5 is a schematic diagram, portions of Figure 5, such as screen image 300, show the approximate locations and sizes of certain items, such as heart pump type 306 and graphs 318 and 320, that are consistent with the exemplary virtual display screen content 200 and virtual display screen content 400 shown in Figures 2 and 4. Figure 5 generally illustrates a process for processing a screen display image 300, such as virtual display screen content 200 of Figure 2 or virtual display screen content 400 of Figure 4, regardless of the differences between virtual display screen content 200 and virtual display screen content 400.

[0059] Aspects of the present invention solve the problem by creating copies of each source region that appear at different screen coordinates on the screen 104 of each of the different medical device controllers 100. Each of these copies is processed by the OCR engine 302, potentially generating multiple text results, e.g., Text5 502 and Text6 504. After the mosaic image 506 is processed by the OCR engine 302, one of the text results Text5 502 or Text6 504 is selected, based on which medical device controller 100 generated the screen image 300. Which medical device controller 100 generated the screen image can be determined, for example, by a software version number 508 displayed on the screen 104 and processed by the OCR engine 302, as described in detail below.

[0060] Continuing with exemplary virtual display screen content 200 and virtual display screen content 400, as noted above, positioning signal 207 (FIG. 2) appears at different screen coordinates in display screen content 200 than positioning signal 407 (FIG. 4) in display screen content 400. As shown in FIG. 5, positioning signal 207 (from FIG. 2) appears in source region 510, while positioning signal 407 (from FIG. 4) appears in source region 512. Source region 512 overlaps source region 512. Each of these source regions 510 and source region 512 is copied into separate predefined destination regions 514 and 516, respectively, in mosaic image 506. Ideally, destination regions 514 and 516 do not overlap.

[0061] Similarly, current blood flow rate 212 (FIG. 2) also appears at different screen coordinates in display screen content 200 than current blood flow rate 412 (FIG. 4) in display screen content 400. As shown in FIG. 5, current blood flow rate 212 (from FIG. 2) appears in source region 310, as seen in FIG. 3, while current blood flow rate 412 (from FIG. 4) appears in source region 518. Each of these source regions 310 and 518 is copied into separate destination regions 520 and 522, respectively, in mosaic image 506. Ideally, destination regions 520 and 522 do not overlap.

[0062] Source region 310 overlaps source region 518. However, overlap between the two source regions is not necessary. Other text and / or graphics in close proximity to one or both of source regions 310 and / or 518 can be avoided by copying source regions 310 and 518 into separate destination regions 520 and 522.

[0063] Source regions that do not cause problems for OCR engine 302 are each copied to just one respective destination region. For example, if heart pump type 202 (FIG. 2) occupies the same screen coordinates as heart pump type 404 (FIG. 4), then the source region containing the heart pump type text on screen 104 is copied to a single destination region 526 in mosaic image 506. If heart pump type 202 (FIG. 2) does not occupy the same screen coordinates as heart pump type 404 (FIG. 4), but a single larger source region can still be copied to a single destination region, without risk of OCR errors.

[0064] Similarly, source region 508 can be copied to a single destination region 528 if the software version numbers occupy the same screen coordinates in both virtual display screen content 200 and virtual display screen content 400, or if a larger source region can encompass both software version numbers without risk of OCR errors.

[0065] The coordinates of the various source regions 310, 508, 510, 512, 518 and 524 and the coordinates of the various destination regions 514, 516, 520, 522, 526 and 528 may be stored in a configuration file rather than being hard-coded to facilitate adding and adjusting these values ​​in response to software changes performed by the medical device controller 100, for example.

[0066] Once the mosaic image 504 is generated, it is passed to the OCR engine 302 for conversion to text. The OCR engine 302 returns text results 530 represented by Text1 532, Text2 534, Text3 536, Text4 538, Text5 502, and Text6 504, which correspond to the target regions 514, 516, 528, 526, 520, and 522, respectively, as discussed herein with respect to FIG. 3. The server extracts Text1 532, Text2 534, ..., Text6 504 from the text results 530.

[0067] Assuming that the differences between virtual display screen content 200 and virtual display screen content 400 are attributable to the versions of software running on the respective medical device controllers 100 that generated screen content 200 and screen content 400, the server extracts Text3 536 from text result 530. Text3 536 includes optical character recognition of the software version number displayed on screen 104, for example, as shown at 220 (FIG. 2). Selector 540 selects either Text5 502 or Text6 504 depending on the value of Text3 536, i.e., the software version. For example, selector 540 may compare Text3 536 to possible values ​​of the software version and then select either Text5 502 or Text6 504 depending on which possible value of the software version matches the actual software version value Text3 536. Thus, Text3 536 serves as input 542 to selector 540. The text selected by selector 540 is provided as output 544 from selector 540 .

[0068] The selected text 544 may then be processed by another component, such as a comparator, which compares the value with predetermined minimum and maximum values ​​and raises an alarm if the current value is outside the range defined by the minimum and maximum values.

[0069] Similarly, selector 540 selects between text1 532 and text2 534, which correspond to destination area 514 and destination area 516, respectively. While the example in Figure 5 illustrates a selection from only two options, more than two types of medical device controllers 100 may transmit screen images 300. In this case, each of some of source areas 310, 508-512, 518, and 524 may be copied to three, four, or more separate destination areas in mosaic image 506, as needed to avoid optical character recognition ambiguities.

[0070] In the foregoing example, the version number of the software being executed by the medical device controller 100 is used to select between the duplicated destination regions 520, 522 and the duplicated destination regions 514, 516. However, in other cases, other information that can be extracted from the display image 300 may be used to provide input 542 to the selector 540. For example, if appropriate, the heart pump type 202 (FIG. 2) encompassed by the source region 524 and the destination region 526 and corresponding to the text 4 538 may be used as input 542 to the selector 540.

[0071] Preferably, the text or other indicator from the display image 300 used to derive the input 542 to the selector 540 resides at the same screen coordinates on each screen 104 of the various medical device controllers 100. However, as with the software version number, the text or other indicator may be located at different coordinates on the screens 104 of the different medical device controllers 100, as exemplified by virtual display screen content 200 (FIG. 2) and virtual display screen content 400 (FIG. 4). In some cases, the source region used to drive the selector 540, e.g., source region 508, need not encompass all possible screen locations where the text or other indicator may be displayed, as long as the selector 540 receives enough input 542 to make a selection.

[0072] For example, if source region 508 includes software version 220 (FIG. 2) in virtual display screen content 200 but does not include software version 422 (FIG. 4) in virtual display screen content 400, then if text3 536 includes "X7.1" (the software version number displayed in virtual display screen content 200), selector 540 may select text5 502, and if text3 536 does not include "X7.1," selector 540 may select text6 504. In other words, if input 542 does not match any predetermined value, selector 540 will default to a predetermined selection, such as text3 536.

[0073] In this example, only target region 528 is used to drive selector 540, but input 542 to selector 540 may include text from any combination of target regions 514, 516, 520, 522, 526, and 528. For example, if the software version number is greater than a particular value and the heart pump is a particular model, an OCR value derived from one target region may be used, but if the software version number is less than or equal to the particular value, an OCR value derived from a different target region may be used. Furthermore, the decision process performed by selector 540 may include several steps. For example, if the medical device is a particular heart pump model, the software version number may be used to make the selection, but if the medical device is another particular heart pump model, the heart pump serial number may be used to make the selection.

[0074] Examples of text or other indicators from the display image 300 that may be used to derive input 542 to the selector 540 include a software version, a hardware version, a heart pump model identifier, a heart pump serial number, an indication of optional features or components present or installed on the heart pump or medical device control device, an indication of options selected by a user or administrator (e.g., to customize how information is displayed on the screen 104), and combinations thereof.

[0075] FIG. 6 is a schematic block diagram of the major components of a medical device monitoring system 600 for collecting, storing, and retrieving operational data from and relating to multiple medical device controllers 100. For simplicity, FIG. 6 shows only the medical device controllers 100, not the individual medical devices. Although three medical device controllers 100 are shown, other numbers of medical device controllers 100 may be used. Each medical device controller 100 is optionally connectable to a computer network 602 via a remote link module 604. Each medical device controller 100 is configured to automatically and repeatedly retrieve status information about the connected medical devices and display the status information on the display screen 104 (FIG. 1). As noted above, FIGS. 2 and 4 illustrate hypothetical display screen contents 200 and 400, respectively, that may be displayed on the screen 104 of any given medical device controller 100.

[0076] The server 606 is configured to automatically, periodically, or on an ad-hoc basis, typically about every 20 seconds, request and receive images of the content displayed on the display screen 104 of each medical device controller 100. The requests and images are transmitted over the computer network 602. The images may be transmitted in one or more messages encoded as a video frame or a sequence of video frames. The video frames may, for example, include a pixelated copy of the image displayed on the display screen 104 of the medical device controller 100.

[0077] The server 606 is configured to process the received frames (images) as described with respect to Figures 2-5. As described above, validation rules, organization procedures, and coordinates of the various source regions 310, 508, 510, 512, 518, and 524, as well as coordinates of the various destination regions 514, 516, 520, 522, 526, and 528, may be stored in a configuration file 607. The server 606 may communicate with an external OCR engine (not shown) via the computer network 602. The server 606 may then use the recognized, and possibly selected, text to automatically verify the serial numbers or other identifiers of the medical device controllers 100, the operating parameters of the medical device controllers 100, whether an alarm has been issued by one of the medical device controllers 100, etc.

[0078] The data store 608 is configured to store frames (images) of one or more media files, particularly MP4 videos or other suitable types of media files, and the server 606 is configured to automatically store received frames (images) in the data store 608. The data store 608 records screen images received by the server 606 for later playback, such as in response to a request from one of several monitoring stations 610. The data store 608 is configured to provide requested portions of stored media files in response to a request to provide them. The data store 608 thereby supports playback of status information for the medical device controller 100. For example, the data store 608 may provide one or more frames (images) of a video stored in a media file for display to a user.

[0079] The server 606 may also be configured to provide status information regarding one or more of the medical device control devices 100 to more than one of the monitoring stations 610 based on images received by the server 606 in real time and / or based on historical information maintained in the data store 608.

[0080] Figure 7 is a schematic block diagram of server 606 of Figure 6. Server 606 includes a mosaic image generator 700 configured to generate mosaic image 506 (Figure 5). Mosaic image generator 700 includes a first source region copier 704, a second source region copier 706, and an additional source region copier 708. Although the first, second, and additional source region copiers 704-708 are shown as separate components, some or all of these copiers 704-708 may share code or other components.

[0081] For each piece of information that appears at different screen coordinates in virtual display screen content 200 and virtual display screen content 400 and that may cause an OCR error if extracted as a single source region from screen image 300 ( FIG. 5 ), the first source region copier is configured to copy one of the source regions encompassed by the first screen coordinate in screen image 300 to a predefined destination region at a third coordinate in mosaic image 506, as discussed herein. Similarly, the second source region copier is configured to copy the other source region encompassed by the second screen coordinate in screen image 300 to a predefined destination region at a fourth coordinate in mosaic image 506 that is different from the third coordinate, as discussed herein.

[0082] Examples of such source regions include the current blood flow source regions 310, 518, which are copied into regions of interest 520, 522, respectively, and the location signal source regions 510, 512, which are copied into regions of interest 514, 516, respectively.

[0083] Additional source region copier 708 is configured to copy source regions of screen image 300 that should not cause OCR errors, as discussed herein, to mosaic image 506. Additional source region copier 708 is configured to copy multiple additional source regions of screen image 300 to respective additional defined destination regions in mosaic image 506. Examples of such source regions include pump type source region 524 and software version number source region 508, which are copied to destination region 526 and destination region 528, respectively.

[0084] As noted above, the server 606 may include an OCR engine, or the server 606 may make a service call to an external OCR engine. Both of these aspects are represented by the OCR engine 302 in Figures 5 and 7. After the mosaic image 506 is generated, the OCR engine parses the mosaic image 506 and returns text results 530.

[0085] The text extractor 710 is configured to extract a first text result 712 from the text results 530. The first text result 712 corresponds to a third coordinate in the mosaic image 506. For example, the first text result 712 may correspond to the current blood flow source region 310 and the corresponding destination region 520.

[0086] The text extractor 710 is also configured to extract a second text result 714 from the text result 530. The second text result 714 corresponds to a fourth coordinate in the mosaic image 506. For example, the second text result 714 may correspond to the current blood flow source region 518 and the corresponding destination region 522.

[0087] The text extractor 710 is further configured to extract at least one additional text result 716 from the text results 530. The at least one additional text result 716 corresponds to at least one additional defined region of interest within the mosaic image 506. For example, the at least one additional text result 716 may correspond to the pump type source region 524 corresponding to the region of interest 526, and / or the at least one additional text result 716 may correspond to the software version number source region 508 corresponding to the region of interest 528.

[0088] The text selector 540 selects either the first text result 712 or the second text result 714 based on the additional text result 716. For example, the text selector 540 may select the first text result 712 or the second text result 714 based on the software version or based on the type of heart pump. The text selector 540 provides selected text 544.

[0089] As described above, the server 606 may expand a particular source region. Optionally, a source region expander 718 may be configured to expand a first source region to fill a defined destination region at a third coordinate.

[0090] As described above, the server 606 may convert colored pixels to white pixels. Optionally, a color pixel whitener 720 may be configured to convert each colored pixel in the first source region to a white pixel in a defined destination region at a third coordinate.

[0091] FIG. 8 is a schematic flowchart illustrating operations of a method for monitoring medical devices. The method may be performed by server 606 (FIGS. 6 and 7), which may be configured to perform the method. At operation 800, the method includes requesting and receiving images ("received images"). The received images are requested and received via a computer network. The received images are requested and received by each medical device controller of a plurality of medical device controllers. Each received image includes content displayed on a screen of the medical device controller. The plurality of medical device controllers includes a plurality of first medical device controllers, each configured to display a first type of information at a first screen coordinate, and a plurality of second medical device controllers, each configured to display the first type of information at a second screen coordinate different from the first screen coordinate.

[0092] A mosaic image is generated for each received image at 802. Generating each mosaic image includes copying 804 a first source region encompassing a first screen coordinate of the received image to a predefined destination region at third coordinates in the mosaic image, copying 806 a second source region encompassing a second screen coordinate of the received image to a predefined destination region at fourth coordinates in the mosaic image that are different from the third coordinates, and copying 808 a plurality of additional source regions of the received image to respective additional predefined destination regions in the mosaic image.

[0093] After generating the mosaic image, at 810, the mosaic image is optically character recognized in a single service call to generate text results.

[0094] At 812, a first text result is extracted from the text results. The first text result corresponds to a third coordinate in the mosaic image. At 814, a second text result is extracted from the text results. The second text result corresponds to a fourth coordinate in the mosaic image. At 816, at least one additional text result is extracted from the text results. The at least one additional text result corresponds to at least one of the additional defined regions of interest in the mosaic image.

[0095] At 818, one of the first text result and the second text result is selected based on the at least one additional text result.

[0096] 9 is a flowchart that schematically illustrates operation 818 (FIG. 8) of selecting one of the first and second text results based on at least one additional text result, assuming that the selection is based on a version number, with possible version numbers being "V1," "V2," etc. Of course, if the selection criteria are different, the flowchart can be modified accordingly. At 900, additional text 716 (FIG. 7) is compared to "V1." If additional text 716 includes (or is equal to) "V1," control proceeds to 902, where the first text result 712 from OCR engine 302 is selected. Control then proceeds to 904, where the selected text is returned as text result 544.

[0097] On the other hand, if at least one additional text result does not contain (or is not equal to) "V1" at operation 900, control proceeds to 906, where additional text 716 is compared to "V2." If additional text 716 contains (or is equal to) "V2," control proceeds to 908, where the second text result 714 from OCR engine 302 is selected. Control then proceeds to 904, where the selected text is returned as text result 544.

[0098] Additional checks may be performed to compare the additional text 716 to other predetermined values, as indicated by ellipsis 910. However, if no match is found in any of the comparisons 900, 906, etc., control proceeds to 912 to discard the received image 300 (FIG. 5).

[0099] Figure 10 is a flowchart that generally illustrates the operations performed by any color pixel whitener 720 (Figure 7). At 1000, a loop is begun. The loop is traversed once for each pixel (called the "current pixel"). At 1002, if the current pixel is not black (or the sum of the pixel's red, green, and blue values ​​exceeds a predetermined threshold), control proceeds to 1004, where the pixel's red value is set to its maximum possible value, such as FF (hex). From there, control proceeds successively to 1006 and 1008, where the pixel's green value is set to its maximum possible value and the pixel's blue value is set to its maximum possible value, respectively. Control then proceeds to 1010.

[0100] On the other hand, if at 1002 the current pixel is black (or the sum of the red, green, and blue values ​​of the pixel is below a predetermined threshold), control passes to 1010, where it is determined whether there are more pixels to process in the image. If there is at least one more pixel to process, control passes to 1012, where the current pixel is advanced to the next pixel, and then the loop returns control to 1002. On the other hand, if all pixels have been processed, control passes from 1010 to 1014.

[0101] While the present invention has been described through the above exemplary embodiments, modifications and variations to the exemplary embodiments may be made without departing from the inventive concepts disclosed herein. For example, although specific parameter values ​​such as display screen field names (e.g., heart pump type, deployment signal, blood flow rate, software version number, and heart pump serial number) may be cited in connection with the disclosed embodiments, the values ​​of all parameters may vary widely to suit different applications within the scope of the present invention. For example, fields other than the software version number and / or heart pump type may be used to distinguish between screen images generated by different medical device controllers.

[0102] As used herein, including the appended claims, the term "and / or" when used in connection with a list of items means one or more of the items in the list, i.e., at least one of the items in the list, but not necessarily all of the items in the list. As used herein, including the appended claims, the term "or" when used in connection with a list of items means one or more of the items in the list, i.e., at least one of the items in the list, but not necessarily all of the items in the list. "Or" does not mean "exclusive or."

[0103] Although aspects of the embodiments may be described with reference to flowcharts and / or block diagrams, all or some of the functions, operations, decisions, etc. of each block, or combinations of blocks, may be combined, separated into separate operations, or performed in other orders. References to "modules," "operations," "steps," and similar terms are for convenience and are not intended to limit the embodiments. All or some of each block, module, operation, step, or combination thereof may be implemented as computer program instructions (e.g., software), hardware (e.g., combinational logic, application-specific integrated circuit (ASIC), field programmable gate array (FPGA), processor, or other hardware), firmware, or a combination thereof.

[0104] Server 606, or portions thereof, may be implemented by one or more processors that execute or are controlled by instructions stored in memory, where each processor may be a general-purpose processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a special-purpose processor, or the like, or a combination thereof, as appropriate.

[0105] The memory may be random access memory (RAM), read-only memory (ROM), flash memory, or any other memory, or combination thereof, suitable for storing control software or other instructions and data. Instructions defining the functionality of the present invention may be provided to a processor in many forms, including, but not limited to, information persistently stored on a tangible, non-transitory, non-writable storage medium (e.g., a read-only memory device within the computer such as a ROM, or a device readable by a computer I / O attachment such as a CD-ROM or DVD disk), information revocably stored on a tangible, non-transitory, writable storage medium (e.g., a floppy disk, removable flash memory, and hard drive), or information transmitted to a computer via a communications medium, including a wired or wireless computer network. Furthermore, although aspects may be described with reference to various exemplary data structures, the system may be embodied using a variety of data structures.

[0106] Aspects of the disclosure, or portions thereof, may be combined in ways not described above and / or not expressly claimed. In addition, the embodiments disclosed herein may suitably be practiced in the absence of any element not specifically disclosed herein. Thus, the present invention should not be considered limited to the disclosed embodiments.

[0107] As used herein, numerical terms such as "first," "second," and "third" are used to distinguish respective source and destination regions from one another and are not intended to indicate any particular order or total number of source or destination regions in any particular embodiment. Thus, for example, a given embodiment may include only a second and a third destination region.

Claims

1. A medical device monitoring system including a server, the server comprising: Automatically, requesting and receiving an image of the content displayed on the screen of the medical device control device from the medical device control device via a computer network; generating a mosaic image, wherein the server: (a) copying a first source region encompassing a first screen coordinate of the received image to a predefined destination region at a fourth coordinate in the mosaic image; (b) copying a second source region of the received image that encompasses a second screen coordinate to a predefined destination region in the mosaic image at a fifth coordinate that is different from the fourth coordinate; (c) copying a third source region encompassing a third screen coordinate of the received image to a predefined destination region in the mosaic image at a sixth coordinate different from the fourth and fifth coordinates; It is configured as follows: After the mosaic image is generated, the mosaic image is subjected to optical character recognition to generate a text result, wherein the text result comprises: (a) a first text result corresponding to the fourth coordinate within the mosaic image; (b) a second text result corresponding to the fifth coordinate within the mosaic image; and (b) a selector text result corresponding to the sixth coordinate within the mosaic image. Including, selecting one of the first text result or the second text result based on the selector text result; 1. A medical device monitoring system configured to:

2. The medical device monitoring system of claim 1 , wherein the selector text result includes a version number of software executed by the medical device controller.

3. The medical device monitoring system of claim 1 , wherein the selector text result includes an identifier of a medical device coupled to the medical device controller.

4. The medical device monitoring system of claim 3 , wherein the identifier of the medical device comprises a serial number or a type code.

5. The medical device monitoring system of claim 1 , wherein the selector text result includes an identifier of an optional feature installed on the medical device control device.

6. the defined destination region at the fourth coordinate in the mosaic image is larger than the first source region; to copy the first source area into the defined destination area at the fourth coordinate, the server is configured to expand the first source area to fill the defined destination area at the fourth coordinate. The medical device monitoring system of claim 1 .

7. 2. The medical device monitoring system of claim 1, wherein the server is configured to convert at least one non-black pixel of the first source region to a white pixel in the defined destination region at the fourth coordinate to copy the first source region to the defined destination region at the fourth coordinate.

8. 1. A method for monitoring a medical device, comprising: requesting and receiving, via a computer network, from a medical device control device, an image of the content displayed on the screen of the medical device control device; generating a mosaic image, said generating said mosaic image comprising: (a) copying a first source region encompassing a first screen coordinate of the received image to a predefined destination region at a fourth coordinate in the mosaic image; (b) copying a second source region of the received image that encompasses a second screen coordinate to a predefined destination region in the mosaic image at a fifth coordinate that is different from the fourth coordinate; (c) copying a third source region encompassing a third screen coordinate of the received image to a predefined destination region at a sixth coordinate in the mosaic image, the sixth coordinate being different from the fourth and fifth coordinates; and After generating the mosaic image, subjecting the mosaic image to optical character recognition to generate a text result, the text result comprising: (a) a first text result corresponding to the fourth coordinate within the mosaic image; (b) a second text result corresponding to the fifth coordinate within the mosaic image; and (b) a selector text result corresponding to the sixth coordinate within the mosaic image. and selecting one of the first text result or the second text result based on the selector text result; A method comprising:

9. The method of claim 8 , wherein the selector text result includes a version number of software executed by the medical device controller.

10. The method of claim 8 , wherein the selector text result includes an identifier of a medical device coupled to the medical device controller.

11. The method of claim 10 , wherein the identifier of the medical device comprises a serial number or a type code.

12. The method of claim 8 , wherein the selector text result includes an identifier of an optional feature installed on the medical device controller.

13. the defined destination region at the fourth coordinate in the mosaic image is larger than the first source region; copying the first source region into the defined destination region at the fourth coordinate includes expanding the first source region to fill the defined destination region at the fourth coordinate. The method of claim 8.

14. 9. The method of claim 8, wherein copying the first source region to the defined destination region at the fourth coordinate includes converting at least one non-black pixel of the first source region to a white pixel in the defined destination region at the fourth coordinate.

15. 1. A non-transitory computer-readable medium encoded with instructions that, when executed by a processor, establish a process for performing a computer-implemented method for monitoring a medical device, the process comprising: a process configured to request and receive, via a computer network, from a medical device controller, an image of the content displayed on a screen of said medical device controller; 1. A process configured to generate a mosaic image, said generating said mosaic image comprising: (a) copying a first source region encompassing a first screen coordinate of the received image to a predefined destination region at a fourth coordinate in the mosaic image; (b) copying a second source region of the received image that encompasses a second screen coordinate to a predefined destination region in the mosaic image at a fifth coordinate that is different from the fourth coordinate; (c) copying a third source region encompassing a third screen coordinate of the received image to a predefined destination region at a sixth coordinate in the mosaic image, the sixth coordinate being different from the fourth and fifth coordinates; a process comprising: a process configured to generate the mosaic image, and then subject the mosaic image to optical character recognition to generate a text result, the text result comprising: (a) a first text result corresponding to the fourth coordinate within the mosaic image; (b) a second text result corresponding to the fifth coordinate within the mosaic image; and (b) a selector text result corresponding to the sixth coordinate within the mosaic image. a process comprising: a process configured to select one of the first text result or the second text result based on the selector text result; 1. A non-transitory computer-readable medium comprising:

16. 2. The medical device monitoring system of claim 1, wherein the server is configured to convert each non-black pixel of the first source region to a white pixel in the defined destination region at the fourth coordinate to copy the first source region to the defined destination region at the fourth coordinate.

17. 17. The medical device monitoring system of claim 16, wherein to copy the second source region to the defined destination region at the fifth coordinate, the server is configured to convert each non-black pixel of the second source region to a white pixel in the defined destination region at the fifth coordinate.

18. 17. The medical device monitoring system of claim 16, wherein to copy the third source region to the defined destination region at the sixth coordinate, the server is configured to convert each non-black pixel of the third source region to a white pixel in the defined destination region at the sixth coordinate.

19. The medical device monitoring system of claim 1 , wherein the first source region overlaps the second source region of the received image.

20. 20. The medical device monitoring system of claim 19, wherein the defined region of interest at the fourth coordinate in the mosaic image does not overlap with the defined region of interest at the fifth coordinate in the mosaic image.

Images