Intelligent image segmentation prior to optical character recognition (OCR)

Abstract

To provide: a system capable of intelligently segmenting images of screens displayed by a medical device controller before subjecting the images to optical character recognition (OCR) processing; a method; and a computer readable medium.SOLUTION: A medical device monitoring method includes the steps of: extracting information from screen images from medical device controllers, with a single OCR process invocation per screen image, despite critical information appearing in different screen locations, depending on which medical device controller's screen image is processed; creating a mosaic image one copy at a time from each screen position with different important screen information; subjecting the mosaic image to OCR processing to produce text results; and selectively extracting text from the OCR text results according to the contents of a selector field on the screen image such as a software version number or a heart pump model identifier.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Patent Application No. 16 / 360,270, filed on March 21, 2019, entitled "Intelligent Image Segmentation Before Optical Character Recognition (OCR)", the entire content of which is incorporated herein by reference for all purposes.

[0002] Technical Field The present invention relates to remotely monitoring medical devices, and more particularly, to intelligently segmenting an image of a screen displayed by a medical device controller before subjecting the image to optical character recognition (OCR) processing.

Background Art

[0003] Related Art Many medical devices, such as some intravascular blood pumps, e.g., the Impella® 2.5 heart pump available from Abiomed, Inc. of Danvers, Massachusetts, are connected to an external medical device controller that collects and displays operating data regarding the medical device, such as heart signal levels, battery temperature, blood flow rate, and tubing integrity. An exemplary medical device controller is available from Abiomed, Inc. under the trade name Automated Impella Controller®. These medical device controllers issue an alarm when an operating data value exceeds a predetermined value or range, e.g., when a leak or loss of suction is detected. These medical device controllers include a video display screen as a human interface, on which the operating data and / or alarms are displayed.

[0004] To facilitate remote monitoring by healthcare providers and ensure effectiveness and patient safety, some of such medical device control devices can be coupled to a central server that can be accessed by a monitoring station via a computer network that often includes a wireless segment. The monitoring station can display real-time operating data and / or alarms on a display screen for healthcare providers to view.

[0005] The server requests and receives an image of the content displayed on the screen of the medical device control device. Some servers use optical character recognition (OCR) technology to parse the image and extract text information such as the heart pump serial number, blood flow rate, warning message text, etc. To reduce OCR errors, some servers mask portions of the screen image that contain irrelevant text or graphics that could potentially confuse the OCR engine.

[0006] However, different medical device control devices may display the same type of information, such as blood flow rate, at different positions on each screen, depending on, for example, which version of software the medical device control device is running or which type of heart pump is connected to the medical device control device. This uncertainty in screen position makes it difficult or impossible to effectively mask the appropriate portion of the screen image before OCR. Thus, the server may use multiple masks, perform the OCR process with each mask, and then select one OCR result. However, OCR processing is costly and time-consuming. The provider would prefer to have the server perform only one OCR process per screen image. Unfortunately, the server has no predictive ability and thus cannot determine in advance which mask to use. SUMMARY OF THE INVENTION

[0007] SUMMARY OF ASPECTS One aspect of the present invention provides a medical device monitoring system. This system includes a server. The server is configured to automatically request and receive a "received image". The server is configured to request and receive the received image via a computer network. The server is configured to request and receive the received image from each of a plurality of medical device control devices. Each received image includes the content displayed on the screen of the medical device control device. The plurality of medical device control devices include a plurality of first medical device control devices and a plurality of second medical device control devices. For example, the first medical device control device may execute a specific version of software, and the second medical device control device may execute a different version of software. In another example, different types of medical devices are connected to the first medical device control device and the second medical device control device, respectively.

[0008] Each first medical device control device is configured to display information of a first type at first screen coordinates. Each second medical device control device is configured to display information of the first type at second screen coordinates different from the first screen coordinates. In other words, the first medical device control device and the second medical device control device display certain information (the "first information"), such as the minimum blood flow rate, at different positions on their respective screens. This difference in screen position may be due to a difference in software version or different types of medical devices connected to the first medical device control device and the second medical device control device.

[0009] For each received image, the server is configured to automatically generate a mosaic image. To generate the mosaic image, the server is configured to copy a first source region of the received image to a defined target region at a third coordinate within the mosaic image. The first source region encompasses the first screen coordinates of the received image. The server is also configured to copy a second source region of the received image to a defined target region at a fourth coordinate within the mosaic image. The fourth coordinate is different from the third coordinate. The second source region encompasses the second screen coordinates of the received image. The server is further configured to copy a plurality of additional source regions of the received image to respective additional defined target regions within the mosaic image.

[0010] After the mosaic image is generated, the server is configured to automatically perform optical character recognition on the mosaic image in a single service call to generate a text result.

[0011] The server is also configured to automatically extract a selector text result from the text result. The selector text result corresponds to at least one of the additional defined target regions within the mosaic image. The server is configured to compare the selector text result with a first defined text. If it is found that the selector text result matches the first defined text, the server is configured to extract the selected text result from the text result such that the selected text result corresponds to the third coordinate within the mosaic image. Otherwise, the server is configured to extract the selected text result from the text result such that the selected text result corresponds to the fourth coordinate within the mosaic image. The server is configured to provide the selected text result.

[0012] Thus, the medical device control device can display the same information at different screen positions, but the server can select the OCR result from the appropriate one of the screen positions based on the content of another field on the screen without the need to perform two OCR processes.

[0013] In any aspect, if the server determines that the selector text result does not match the first predefined text, the server may be configured to automatically compare the selector text result with a second predefined text. The second predefined text is different from the first predefined text. If it is determined that the selector text result matches the second predefined text, the server may be configured to automatically extract the selected text result from the text results such that the selected text result corresponds to a fourth coordinate within the mosaic image.

[0014] In any aspect, at least one of the additional predefined target regions within the mosaic image may correspond to an additional source region that includes a version number of software executed by a medical device control device.

[0015] In any aspect, at least one of the additional predefined target regions within the mosaic image may correspond to an additional source region that includes an identifier of a medical device coupled to a medical device control device.

[0016] In any aspect, the identifier of the medical device may include a serial number.

[0017] In any aspect, the identifier of the medical device may include a type code.

[0018] In any aspect, at least one of the additional predefined target regions within the mosaic image may correspond to an additional source region that includes an identifier of an optional function installed in a medical device control device.

[0019] In any aspect, the predefined target region at the third coordinate within the mosaic image may be larger than the first source region. To copy the first source region to the predefined target region at the third coordinate, the server may be configured to expand the first source region to fill the predefined target region at the third coordinate.

[0020] In any aspect, to copy a first source region to a defined target region at a third coordinate, the server may be configured to convert each colored pixel of the first source region to a white pixel within the defined target region at the third coordinate.

[0021] Another aspect of the present invention provides a method for monitoring medical devices. The method includes the step of requesting and receiving a received image. The requested image is requested and received via a computer network. The received image is requested and received from each medical device control device of a plurality of medical device control devices. Each received image includes the content displayed on the screen of the medical device control device. The plurality of medical device control devices include a plurality of first medical device control devices and a plurality of second medical device control devices. Each first medical device control device is configured to display information of a first type at a first screen coordinate, and each second medical device control device is configured to display information of the first type at a second screen coordinate different from the first screen coordinate.

[0022] The method includes the step of generating a mosaic image for each received image. The step of generating a mosaic image includes copying a first source region including the first screen coordinates of the received image to a defined target region at a third coordinate within the mosaic image. A second source region including the second screen coordinates of the received image is copied to a defined target region at a fourth coordinate different from the third coordinate within the mosaic image. A plurality of additional source regions of the received image are copied to respective additional defined target regions within the mosaic image.

[0023] After generating the mosaic image, the mosaic image is optically character recognized in a single service call to generate a text result. A selector text result is extracted from the text result. The selector text result corresponds to at least one of the additional defined target regions within the mosaic image. The selector text result is compared with a first defined text.

[0024] If it is found that the selector text result matches the first predefined text, the selected text result is extracted from the text result such that the selected text result corresponds to the third coordinates in the mosaic image. Otherwise, the selected text result is extracted from the text result such that the selected text result corresponds to the fourth coordinates in the mosaic image. Then, the selected text result is provided.

[0025] In any aspect, when not so, the step of extracting the selected text result may include comparing the selector text result with a second predefined text. The second predefined text may be different from the first predefined text. If it is found that the selector text result matches the second predefined text, the selected text result is extracted from the text result such that the selected text result corresponds to the fourth coordinates in the mosaic image.

[0026] In any aspect, at least one of the additional predefined target regions in the mosaic image may correspond to an additional source region including the version number of the software executed by the medical device control device.

[0027] In any aspect, at least one of the additional predefined target regions in the mosaic image may correspond to an additional source region including the identifier of the medical device coupled to the medical device control device.

[0028] In any aspect, the identifier of the medical device may include a serial number.

[0029] In any aspect, the identifier of the medical device may include a type code.

[0030] In any aspect, at least one of the additional predefined target regions in the mosaic image may correspond to an additional source region including the identifier of the optional function installed in the medical device control device.

[0031] In any aspect, a defined target region at a third coordinate within a mosaic image may be larger than the first source region. Copying the first source region to the defined target region at the third coordinate may include expanding the first source region to fill the defined target region at the third coordinate.

[0032] In any aspect, copying the first source region to the defined target region at the third coordinate may include converting each colored pixel of the first source region to a white pixel within the defined target region at the third coordinate.

[0033] Yet another aspect of the present invention provides a non - transitory computer - readable medium encoded with instructions. When executed by a processor, the instructions establish a process for performing a computer - implemented method of monitoring medical devices. The process includes a process configured to request and receive, via a computer network, a received image of the content displayed on the screen of each of a plurality of medical device controllers from each of the plurality of medical device controllers. The plurality of medical device controllers include a plurality of first medical device controllers each configured to display a first type of information at a first screen coordinate, and a plurality of second medical device controllers each configured to display the first type of information at a second screen coordinate different from the first screen coordinate.

[0034] The process also includes a process configured to generate a mosaic image for each received image. The process is configured to copy a first source region that encompasses the first screen coordinate of the received image to a defined target region at a third coordinate within the mosaic image. The process is also configured to copy a second source region that encompasses the second screen coordinate of the received image to a defined target region at a fourth coordinate within the mosaic image that is different from the third coordinate. The process is further configured to copy a plurality of additional source regions of the received image to respective additional defined target regions within the mosaic image.

[0035] The process further includes a process configured to perform optical character recognition on the mosaic image in a single service call to generate text results after generating the mosaic image.

[0036] The process includes a process configured to extract selector text results from the text results. The selector text results correspond to at least one of the additional defined target regions in the mosaic image. The process includes a process configured to compare the selector text results with first defined text.

[0037] If it is found that the selector text results match the first defined text, the process includes extracting the selected text results from the text results such that the selected text results correspond to a third coordinate in the mosaic image; otherwise, the process includes extracting the selected text results from the text results such that the selected text results correspond to a fourth coordinate in the mosaic image.

[0038] The process also includes a process configured to provide the selected text results.

Brief Description of the Drawings

[0039] The present invention will be more fully understood by reference to the following detailed description of specific embodiments in conjunction with the drawings.

[0040]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

[0041] DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS Aspects of the present invention provide a medical device monitoring system and a medical device monitoring method that extract information from a screen image from a medical device control device with a single OCR process call for each screen image, even though important information appears at different screen positions depending on which screen image of the medical device control device is processed.

[0042] FIG. 1 is a perspective view of an exemplary conventional medical device controller 100, and an exemplary conventional medical device 102 coupled to the medical device controller 100, which in this example is a heart pump. In the example shown in FIG. 1, the medical device controller 100 is an Automated Impella Controller (registered trademark) manufactured by Abiomed, Inc. of Danvers, Massachusetts, and the heart pump 102 is also an Impella (registered trademark) 2.5 heart pump available from Abiomed, Inc., although any suitable medical device controller may be used. In some cases, the medical device and its associated medical device controller are combined. Such a combination is referred to herein simply as a medical device controller.

[0043] The medical device controller 100 includes a display screen 104, where the medical device controller 100 displays operating data related to the medical device, such as heart signal level, battery temperature, blood flow rate, and tubing integrity. As discussed in more detail herein, the medical device controller 100 is connected to a computer network and can thereby transmit an image of the content displayed on the screen 104 to a remote server (not shown).

[0044] FIG. 2 shows an exemplary virtual display screen content 200 that may be displayed on the screen 104 of the medical device controller 100 of FIG. 1. For example, the display screen content may include a heart pump type ("Impella 5.0") 202, a heart pump serial number ("120703") 204, one or more warning / error / information messages 206, a placement signal 207, a current heart pump speed (performance) setting ("P-0") 208, a heart pump motor current value 210, a current blood flow rate 212, and minimum and maximum blood flow rates 213. The display screen content 200 is typically pixelated.

[0045] Figure 3 schematically shows a process for processing a screen display image 300 such as the virtual display screen content 200 of Figure 2. Although Figure 3 is a schematic diagram, the portion of Figure 3 such as the screen image 300 shows the approximate positions and sizes of specific items such as the source region 306 of the heart pump type, the graph 318, and the graph 320, which coincide with the exemplary virtual display screen content 200 shown in Figure 2.

[0046] As described above, in order to reduce OCR errors, some servers mask portions of the screen images 200, 300 that include irrelevant text or graphics that may confuse the OCR engine 302. For example, the virtual display screen content 200 (Figure 2) includes the graphs 214 and 216, and legends such as "Impella Flow" 218, which may cause or contribute to OCR errors if included in the input to the OCR engine 302.

[0047] To avoid these errors, a mosaic image 304 is generated only from a predetermined portion of the screen display image 300. The mosaic image 304 may have the same dimensions as the screen display image 300 in pixel units, but it may not. In this example, three source regions 306, 308, and 310 of the screen display image 300 are copied to the respective defined target regions 312, 314, and 316 of the mosaic image 304. However, any number of source regions of the screen display image 300 may be copied to the mosaic image 304.

[0048] Each of the source regions 306 - 310 can be identified by corresponding screen coordinates. As used herein, the term screen coordinates, or simply coordinates, means information that identifies the region of an image, i.e., the position and size of the entire space within which information is displayed or arranged within the image. The region may be rectangular, but it may not be. Coordinates can include, for example, the x and y pixel positions of the lower left corner of the region, as well as the height and width of the region in pixel units. Other suitable units of measurement and other suitable representations may be used as long as the position and extent of the region are specified. Thus, even if both fields start at the same x and y positions, fields of different widths will have different coordinates.

[0049] The destination regions 312 - 316 may or may not have the same coordinates as their respective corresponding source regions 306 - 310. That is, the destination regions 312 - 316 may or may not have the same starting position and / or size as their respective corresponding source regions 306 - 310. In some embodiments, one or more of the destination regions 312 - 316 are larger than their corresponding source regions 306 - 310.

[0050] In this case, the server enlarges the source regions 312 - 316 such that the enlarged result fills the corresponding destination regions 312 - 316. Exemplary magnifications include, but are not limited to, magnifications of 1.1x, 1.5x, 2x, and 3x. Such magnifications can reduce the likelihood of OCR errors.

[0051] In some embodiments, the server changes the color of a pixel when the pixel is copied from the source regions 306 - 310 to the destination regions 312 - 316. For example, the server can convert a colored pixel to a white pixel to increase contrast and thus reduce OCR errors. The server can convert, for example, a white pixel whose overall luminance, determined by the sum of its red, blue, and green values, exceeds a threshold, and the server can convert all other pixels to black.

[0052] Irrelevant parts of the screen display image 300, such as graph 318 and graph 320 (corresponding to graph 214 and graph 216 in FIG. 2), are not copied to the mosaic image 304. The screen coordinates of the source regions 306 - 310, if included, should be selected to avoid text and / or graphics that may confuse the OCR engine 302 or cause the OCR engine 302 to include false text in the output of the OCR engine 302. The remaining parts of the mosaic image 304 remain blank.

[0053] When the mosaic image 304 is generated, the mosaic image 304 is passed to the OCR engine 302 for text conversion. The server may include the OCR engine 304, or the server may call an external OCR engine such as the OCR engine of the Google Cloud Vision API provided by Google (Alphabet, Inc., Mountain View, California). The OCR engine 302 returns a text result 321 represented by text 1 322, text 2 324, and text 3 326 corresponding to the target regions 312 - 316 respectively. The server extracts text 1 322, text 2 324, and text 3 326 from the text result 321.

[0054] Optionally, the text result 321, or a part thereof, such as text 1, text 2, and / or text 3 of 322 - 326, may be verified using a defined rule such as "The pump serial number must be a 5 - 7 digit number". Such verification may prevent some OCR errors that could corrupt subsequent displays. Also optionally, the text result 321 or a part thereof 322 - 326 may be arranged according to a defined procedure such as "Remove the leading and trailing spaces of the error message". These defined rules and procedures may be stored in a configuration file described in more detail herein.

[0055] As described above, different medical device controllers 100 (FIG. 1) may display the same type of information, such as blood flow rate, at different positions on respective screens 104, for example, depending on which version of software the medical device controller is executing or which type of heart pump 102 is connected to the medical device controller 100. FIG. 4 shows another exemplary virtual display screen content 400 that may be displayed on the screen 104 of a medical device controller 100 that is executing a different version of software than the medical device controller that generated the screen content 200 (FIG. 2), or that has a different type of heart pump 102 connected thereto, such as an Impella routine CP heart pump manufactured by Abiomed, Inc.

[0056] In the exemplary virtual display screen content 400, the heart pump type ("Impella CP") 402 is different from the corresponding heart pump type 202 shown in the screen content 200 (FIG. 2), indicating that a different type of heart pump is connected to the medical device controller that generated the display screen content 400. The heart pump serial number is shown at 404. As can be seen by comparing FIG. 4 with FIG. 2, some of the information within the display content 400 is displayed at different respective positions from the corresponding information within the display content 200. For example, the placement signal 407, the motor current 410, and the current, minimum, and maximum blood flow rates 412 and 413 are each at screen coordinates (position and / or size) from the same information 207, 210, 212, and 213 within the screen content 200 (FIG. 2).

[0057] Obviously, this difference in screen coordinates causes problems for a single server that receives screen images from both medical device controllers and attempts to mask the relevant portions of the screens in the manner discussed with respect to FIG. 3. Defining a source region that is simply large enough to encompass both versions of the display will fail because such a large source region, such as source region 420, will include irrelevant text ("(mmHg)") within the display content 400.

[0058] As shown schematically in FIG. 5, an aspect of the present invention solves this problem. Although FIG. 5 is a schematic diagram, the portion of FIG. 5 such as the screen image 300 corresponds to the exemplary virtual display screen content 200 and virtual display screen content 400 shown in FIGS. 2 and 4, and shows the approximate positions and sizes of specific items such as the heart pump type 306, the graph 318, and the graph 320. FIG. 5 schematically shows the process for processing the screen display image 300 such as the virtual display screen content 200 in FIG. 2 and the virtual display screen content 400 in FIG. 4, regardless of the differences between the virtual display screen content 200 and the virtual display screen content 400.

[0059] An aspect of the present invention solves the problem by creating copies of each source region that appears at different screen coordinates on the respective screens 104 of different medical device control devices 100. Each of these copies is processed by the OCR engine 302 to generate potentially multiple text results, such as text 5 502 and text 6 504. After the mosaic image 506 is processed by the OCR engine 302, one of the text results text 5 502 or text 6 504, based on which the medical device control device 100 generated the screen image 300, is selected. Which medical device control device 100 generated the screen image can be determined, for example, by the software version number 508 displayed on the screen 104 and processed by the OCR engine 302, as will be described in detail below.

[0060] Continuing with the exemplary virtual display screen content 200 and virtual display screen content 400, as described above, the placement signal 207 (FIG. 2) appears at screen coordinates within the display screen content 200 that are different from the placement signal 407 (FIG. 4) within the display screen content 400. As shown in FIG. 5, the placement signal 207 (from FIG. 2) appears in the source region 510, while the placement signal 407 (from FIG. 4) appears in the source region 512. The source region 512 overlaps the source region 512. Each of these source regions 510 and source region 512 is copied to a separate defined target region 514 and defined target region 516 within the mosaic image 506. Ideally, the target region 514 and the target region 516 do not overlap.

[0061] Similarly, the current blood flow 212 (FIG. 2) also appears at screen coordinates within the display screen content 200 that are different from the current blood flow 412 (FIG. 4) within the display screen content 400. As shown in FIG. 5, the current blood flow 212 (from FIG. 2) appears in the source region 310 as seen in FIG. 3, while the current blood flow 412 (from FIG. 4) appears in the source region 518. Each of these source regions 310 and 518 is copied to a separate target region 520 and 522 within the mosaic image 506. Ideally, the target region 520 and the target region 522 do not overlap.

[0062] The source region 310 overlaps the source region 518. However, the overlap of the two source regions is not necessary. Other text and / or graphics adjacent to one or both of the source regions 310 and / or 518 can be avoided by copying the source regions 310 and 518 to separate target regions 520 and 522.

[0063] Source regions that do not cause problems for the OCR engine 302 are each copied to only one respective destination region. For example, if the heart pump type 202 (Figure 2) occupies the same screen coordinates as the heart pump type 404 (Figure 4), the source region that includes the text of the heart pump type on the screen 104 is copied to a single destination region 526 within the mosaic image 506. Even if the heart pump type 202 (Figure 2) does not occupy the same screen coordinates as the heart pump type 404 (Figure 4), but a single larger source region can include both the heart pump type field 202 and the heart pump type field 404 without the risk of OCR errors, the larger source region can be copied to a single destination region.

[0064] Similarly, if the software version numbers occupy the same screen coordinates in both the virtual display screen contents 200 and the virtual display screen contents 400, or if a larger source region can include both software version numbers without the risk of OCR errors, the source region 508 can be copied to a single destination region 528.

[0065] The coordinates of the various source regions 310, 508, 510, 512, 518, and 524, as well as the coordinates of the various destination regions 514, 516, 520, 522, 526, and 528, can be stored in a configuration file rather than being hard-coded, for example, to facilitate the addition and adjustment of these values in response to changes in the software executed by the medical device control device 100.

[0066] When the mosaic image 504 is generated, the mosaic image 504 is passed to the OCR engine 302 for conversion to text. The OCR engine 302 returns a text result 530 represented by text 1 532, text 2 534, text 3 536, text 4 538, text 5 502, and text 6 504 corresponding to the target regions 514, 516, 528, 526, 520, and 522 respectively, as discussed herein with respect to FIG. 3. The server extracts text 1 532, text 2 534, ..., text 6 504 from the text result 530.

[0067] Assuming that the difference between the virtual display screen content 200 and the virtual display screen content 400 is due to the software version of each medical device control device 100 that generated the screen content 200 and the screen content 400, the server extracts text 3 536 from the text result 530. Text 3 536 includes, for example, an optical character recognition of the software version number displayed on the screen 104 as shown in 220 (FIG. 2). The selector 540 selects either text 5 502 or text 6 504 according to the value of text 3 536, i.e., the software version. For example, the selector 540 compares text 3 536 with the possible values of the software version and then selects either text 5 502 or text 6 504 according to which of the possible values of the software version matches the actual software version value text 3 536. Thus, text 3 536 functions as an input 542 to the selector 540. The text selected by the selector 540 is provided as an output 544 from the selector 540.

[0068] The selected text 544 can then be processed by another component, such as a comparator, that compares the value to a predetermined minimum and maximum value and issues an alarm if the current value is outside the range defined by the minimum and maximum values.

[0069] Similarly, selector 540 selects from text 1 532 and text 2 534, corresponding to target region 514 and target region 516 respectively. Although only selection from two options is described in the example of FIG. 5, more than three types of medical device control apparatuses 100 may transmit the screen image 300. In this case, each of some of the source regions 310, 508-512, 518, and 524 may be copied to three, four, or more separate target regions within the mosaic image 506, as necessary to avoid optical character recognition ambiguity.

[0070] In the foregoing example, the version number of the software being executed by the medical device control apparatus 100 is used to select from the target regions 520, 522 to be replicated and the target regions 514, 516 to be replicated. However, in other cases, other information that can be extracted from the display image 300 may be used to provide the input 542 to the selector 540. For example, if appropriate, the heart pump type 202 (FIG. 2) included by the source region 524 and the target region 526 and corresponding to text 4 538 may be used as the input 542 to the selector 540.

[0071] Preferably, the text or other indicators from the display image 300 used to derive the input 542 to the selector 540 are present at the same screen coordinates on each screen 104 of the various medical device control apparatuses 100. However, similar to the case of the software version number, the text or other indicators may be arranged at different coordinates on the screen 104 of different medical device control apparatuses 100, as illustrated by the virtual display screen content 200 (FIG. 2) and the virtual display screen content 400 (FIG. 4). In some cases, the source region used to drive the selector 540, such as the source region 508, need not include all possible screen positions where text or other indicators may be displayed, as long as the selector 540 receives sufficient input 542 to make a selection.

[0072] For example, if the source region 508 includes the software version 220 (FIG. 2) within the virtual display screen content 200 but does not include the software version 422 (FIG. 4) within the virtual display screen content 400, when the text 3536 includes "X7.1" (the software version number displayed on the virtual display screen content 200), the selector 540 may select the text 5502, and when the text 3536 does not include "X7.1", the selector 540 may select the text 6504. In other words, if the input 542 does not match any predetermined value, the selector 540 will make a default predetermined selection such as the text 3536.

[0073] In this example, only the target region 528 is used to drive the selector 540, but the input 542 to the selector 540 may include text from any combination of the target regions 514, 516, 520, 522, 526, and 528. For example, if the software version number is greater than a specific value and the heart pump is a specific model, the OCR value derived from one target region may be used, but if the software version number is less than or equal to a specific value, the OCR value derived from a different target region may be used. Further, the decision-making process executed by the selector 540 may include several steps. For example, if the medical device is a specific heart pump model, the software version number may be used to make a selection, but if the medical device is another specific heart pump model, the heart pump serial number may be used to make a selection.

[0074] Examples of text or other indicators from the display image 300 that can be used to derive the input 542 to the selector 540 include software version, hardware version, heart pump model identifier, heart pump serial number, display of optional functions or components present or installed in the heart pump or medical device controller, display of options selected by the user or administrator (e.g., to customize the way information is displayed on the screen 104), and combinations thereof.

[0075] FIG. 6 is a schematic block diagram of the main components of a medical device monitoring system 600 for collecting, storing, and retrieving operation data related to a plurality of medical device control devices 100 from the plurality of medical device control devices 100. For simplicity, only the medical device control device 100 is shown in FIG. 6, and separate medical devices are not shown. Although three medical device control devices 100 are shown, other numbers of medical device control devices 100 may be used. Each medical device control device 100 can be optionally connected to a computer network 602 via a remote link module 604. Each medical device control device 100 is configured to automatically and repeatedly capture status information regarding the connected medical device and display the status information on a display screen 104 (FIG. 1). As described above, FIGS. 2 and 4 show respective virtual display screen contents 200 and 400 that can be displayed on the screen 104 of any given medical device control device 100.

[0076] The server 606 is configured to request and receive, automatically at regular intervals, or at any time, typically about every 20 seconds, an image of the content displayed on the screen 104 of each medical device control device 100. The request and the image are transmitted via the computer network 602. The image can be transmitted as one or more messages encoded as a video frame or a sequence of video frames. The video frame can include, for example, a pixelated copy of the image displayed on the display screen 104 of the medical device control device 100.

[0077] As described with respect to FIGS. 2-5, server 606 is configured to process received frames (images). As described above, verification rules, sorting procedures, and the coordinates of various source regions 310, 508, 510, 512, 518, and 524, as well as the coordinates of various destination regions 514, 516, 520, 522, 526, and 528, may be stored in configuration file 607. Server 606 may communicate with an external OCR engine (not shown) via computer network 602. Server 606 may then automatically verify, using the recognized and optionally selected text, the serial number or other identifier of medical device controller 100, the operating parameters of medical device controller 100, whether an alarm has been issued by one of medical device controllers 100, etc.

[0078] Data store 608 is configured to store one or more media files, particularly frames (images) such as MP4 videos and other suitable types of media files, and server 606 is configured to automatically store received frames (images) in data store 608. Data store 608 records screen images received by server 606 for later playback, such as in response to a request from one of several monitoring stations 610. Data store 608 is configured to provide the requested portion of a stored media file in response to a provide request. Data store 608 thereby supports the playback of status information of medical device controller 100. For example, data store 608 may provide one or more frames (images) of a video stored in a media file for display to a user.

[0079] Server 606 may also be configured to provide status information regarding one or more of medical device controllers 100 to a plurality of monitoring stations 610 based on images received in real time by server 606 and / or based on historical information held in data store 608.

[0080] FIG. 7 is a schematic block diagram of the server 606 of FIG. 6. The server 606 includes a mosaic image generator 700 configured to generate a mosaic image 506 (FIG. 5). The mosaic image generator 700 includes a first source area copier 704, a second source area copier 706, and an additional source area copier 708. The first, second, and additional source area copiers 704-708 are shown as separate components, but some or all of these copiers 704-708 may share code or other components.

[0081] For each piece of information that appears at different screen coordinates within the virtual display screen content 200 and the virtual display screen content 400 and that may cause an OCR error when extracted as a single source area from the screen image 300 (FIG. 5), the first source area copier is configured to copy one of the source areas included in the first screen coordinates within the screen image 300 to a defined target area at a third coordinate within the mosaic image 506, as discussed herein. Similarly, the second source area copier is also configured to copy the other source area included by the second screen coordinates within the screen image 300 to a defined target area at a fourth coordinate different from the third coordinate within the mosaic image 506, as discussed herein.

[0082] Examples of such source areas include the current blood flow source areas 310, 518 respectively copied to the target areas 520, 522, and the placement signal source areas 510, 512 respectively copied to the target areas 514, 516.

[0083] The additional source area copier 708 is configured to copy the source areas of the screen image 300 that should not cause OCR errors to the mosaic image 506, as discussed herein. The additional source area copier 708 is configured to copy a plurality of additional source areas of the screen image 300 to respective additional defined target areas within the mosaic image 506. Examples of such source areas include the pump type source area 524 and the software version number source area 508, which are copied to the target area 526 and the target area 528, respectively.

[0084] As described above, the server 606 may include an OCR engine, or the server 606 may make a service call to an external OCR engine. Both of these aspects are represented by the OCR engine 302 in FIGS. 5 and 7. After the mosaic image 506 is generated, the OCR engine parses the mosaic image 506 and returns a text result 530.

[0085] The text extractor 710 is configured to extract a first text result 712 from the text result 530. The first text result 712 corresponds to a third coordinate within the mosaic image 506. For example, the first text result 712 may correspond to the current blood flow rate source area 310 and the corresponding target area 520.

[0086] The text extractor 710 is also configured to extract a second text result 714 from the text result 530. The second text result 714 corresponds to a fourth coordinate within the mosaic image 506. For example, the second text result 714 may correspond to the current blood flow rate source area 518 and the corresponding target area 522.

[0087] The text extractor 710 is further configured to extract at least one additional text result 716 from the text result 530. The at least one additional text result 716 corresponds to at least one additional defined target region within the mosaic image 506. For example, the at least one additional text result 716 may correspond to the pump type source region 524 that corresponds to the target region 526, and / or the at least one additional text result 716 may correspond to the software version number source region 508 that corresponds to the target region 528.

[0088] Based on the additional text result 716, the text selector 540 selects either the first text result 712 or the second text result 714. For example, the text selector 540 may select the first text result 712 or the second text result 714 based on the software version or based on the type of the heart pump. The text selector 540 provides the selected (selected) text 544.

[0089] As described above, the server 606 may expand a specific source region. Optionally, the source region expander 718 may be configured to expand the first source region to fill a defined target region at the third coordinate.

[0090] As described above, the server 606 may convert colored pixels to white pixels. Optionally, the color pixel whitenizer 720 may be configured to convert each colored pixel of the first source region to a white pixel within the defined target region at the third coordinate.

[0091] FIG. 8 is a schematic flowchart showing the operation of a method for monitoring a medical device. This method can be performed by a server 606 (FIGS. 6 and 7), and the server 606 can be configured to perform this method. In operation 800, the method includes the step of requesting and receiving an image (the "received image"). The received image is requested and received via a computer network. The received image is requested and received from each of a plurality of medical device control devices. Each received image includes the content displayed on the screen of the medical device control device. The plurality of medical device control devices include a plurality of first medical device control devices each configured to display information of a first type at first screen coordinates, and a plurality of second medical device control devices each configured to display information of the first type at second screen coordinates different from the first screen coordinates.

[0092] At 802, a mosaic image is generated for each received image. The step of generating each mosaic image includes 804 copying a first source region encompassing the first screen coordinates of the received image to a defined target region at a third coordinate within the mosaic image, 806 copying a second source region encompassing the second screen coordinates of the received image to a defined target region at a fourth coordinate within the mosaic image different from the third coordinate, and 808 copying a plurality of additional source regions of the received image to respective additional defined target regions within the mosaic image.

[0093] After generating the mosaic image, at 810, the mosaic image is optically character recognized in a single service call to generate a text result.

[0094] At 812, a first text result is extracted from the text result. The first text result corresponds to the third coordinate within the mosaic image. At 814, a second text result is extracted from the text result. The second text result corresponds to the fourth coordinate within the mosaic image. At 816, at least one additional text result is extracted from the text result. The at least one additional text result corresponds to at least one of the additional defined target regions within the mosaic image.

[0095] In 818, based on at least one additional text result, one of the first text result and the second text result is selected.

[0096] FIG. 9 is a flowchart schematically showing operation 818 (FIG. 8) of selecting one of the first text result and the second text result based on at least one additional text result, assuming that the selection is based on version numbers and possible version numbers are "V1", "V2", etc. Of course, if the selection criteria are different, the flowchart can be modified accordingly. At 900, additional text 716 (FIG. 7) is compared with "V1". If the additional text 716 contains (or is equal to) "V1", the control proceeds to 902, where the first text result 712 from the OCR engine 302 is selected. The control then proceeds to 904, where the selected text is returned as text result 544.

[0097] On the other hand, at operation 900, if at least one additional text result does not contain (or is not equal to) "V1", the control proceeds to 906, where the additional text 716 is compared with "V2". If the additional text 716 contains (or is equal to) "V2", the control proceeds to 908, where the second text result 714 from the OCR engine 302 is selected. The control then proceeds to 904, where the selected text is returned as text result 544.

[0098] As indicated by ellipsis 910, additional inspections can be performed to compare the additional text 716 with other predetermined values. However, if no match is found in any of the comparisons 900, 906, etc., the control proceeds to 912 to discard the received image 300 (FIG. 5).

[0099] FIG. 10 is a flowchart schematically showing the operations performed by an arbitrary color pixel whitener 720 (FIG. 7). At 1000, the loop starts. The loop is traversed once for each pixel (referred to as the "current pixel"). If at 1002 the current pixel is not black (or the sum of the red, green, and blue values of the pixel exceeds a predetermined threshold), control proceeds to 1004, where the red value of the pixel is set to its maximum possible value, such as FF (hexadecimal). From there, control successively proceeds to 1006 and 1008, where the green value of the pixel is set to its maximum possible value and the blue value of the pixel is set to its maximum possible value, respectively. Control then proceeds to 1010.

[0100] On the other hand, if at 1002 the current pixel is black (or the sum of the red, green, and blue values of the pixel is below a predetermined threshold), control proceeds to 1010, where it is determined whether there are still pixels to be processed in the image. If there is at least one more pixel to be processed, control proceeds to 1012, where the current pixel is advanced to the next pixel, and then the loop returns control to 1002. On the other hand, if all pixels have been processed, control proceeds from 1010 to 1014.

[0101] Although the present invention has been described through the above exemplary embodiments, modifications to the exemplary embodiments and variations thereof can be made without departing from the inventive concept disclosed herein. For example, specific parameter values such as display screen field names (e.g., heart pump type, placement signal, blood flow rate, software version number, and heart pump serial number) may be cited in relation to the disclosed embodiments, but within the scope of the present invention, the values of all parameters can vary widely to suit different applications. For example, fields other than the software version number and / or heart pump type may be used to distinguish screen images generated by different medical device control devices.

[0102] As used herein, including in the appended claims, the term "and / or" when used in connection with a list of items means one or more of the items in the list, i.e., at least one of the items in the list, but not necessarily all of the items in the list. As used herein, including in the appended claims, the term "or" when used in connection with a list of items means one or more of the items in the list, i.e., at least one of the items in the list, but not necessarily all of the items in the list. "Or" does not mean "exclusive or".

[0103] Aspects may be described with respect to flowcharts and / or block diagrams. However, all or part of the functions, operations, decisions, etc. of each block, or combination of blocks, may be combined, separated into separate operations, or performed in a different order. References to "module", "operation", "step" and similar terms are for convenience only and are not intended to limit their implementations. All or part of each block, module, operation, step or combination thereof may be implemented as computer program instructions (such as software), hardware (combinational logic, application specific integrated circuit (ASIC), field programmable gate array (FPGA), processor or other hardware), firmware or a combination thereof.

[0104] Server 606 or a portion thereof may be implemented by one or more processors that execute instructions stored in memory or are controlled by instructions stored in memory. Each processor may, as needed, be a general-purpose processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated processor, etc., or a combination thereof.

[0105] The memory can be a random access memory (RAM), read-only memory (ROM), flash memory, or any other memory, or a combination thereof, suitable for storing control software or other instructions and data. The instructions defining the functions of the present invention can be in many forms including, but not limited to, information permanently stored on a tangible non-transitory non-writable storage medium (e.g., a read-only memory device within a computer such as ROM, a device readable by a computer I / O attachment such as a CD-ROM or DVD disk), information changeably stored on a tangible non-transitory writable storage medium (e.g., a floppy disk, removable flash memory, and hard drive), or information transmitted to a computer via a communication medium including a wired or wireless computer network, and can be sent to a processor. Further, although aspects may be described in relation to various exemplary data structures, the system can be embodied using various data structures.

[0106] Aspects of the disclosure, or portions thereof, may be combined in ways not described above and / or not explicitly claimed. Additionally, aspects disclosed herein may be appropriately practiced without elements not specifically disclosed herein. Accordingly, the present invention should not be regarded as limited to the disclosed aspects.

[0107] As used herein, numerical terms such as "first," "second," and "third" are used to distinguish respective source and destination regions from each other and are not intended to indicate any particular order or total number of source or destination regions in any particular aspect. Thus, for example, a given aspect may include only a second and a third destination region.

Claims

1. A medical device monitoring system (600) including a server (606), the server comprising: Automatically, requesting and receiving (800) a received image (300) of content displayed on a screen (104) of a plurality of medical device control devices (100) including a plurality of first medical device control devices each configured to display a first type of information (207 or 212) at a first screen coordinate and a plurality of second medical device control devices each configured to display the first type of information (407 or 412) at a second screen coordinate different from the first screen coordinate via a computer network (602); generating (802) a mosaic image (506) for each received image, where the server: (a) copying (804) a first source region (510, 310) encompassing the first screen coordinate of the received image to a predefined destination region (514, 520) at a third coordinate in the mosaic image; (b) copying (806) a second source region (512, 518) encompassing the second screen coordinate of the received image to a predefined destination region (516, 522) in the mosaic image at a fourth coordinate different from the third coordinate; (c) copying (808) a plurality of additional source regions (508, 524) of the received image to respective additional predefined destination regions (528, 526) in the mosaic image; It is configured as follows: After the mosaic image is generated, subjecting the mosaic image to optical character recognition (810, 302) in a single service call to generate a text result (530); extracting from the text results a selector text result (536 or 538) that corresponds to at least one of the additional defined regions of interest within the mosaic image; comparing said selector text result to a first predefined text; if the selector text result is found to match the first predefined text, extracting a selected text result from the text result such that the selected text (532 or 502) result corresponds to the third coordinate in the mosaic image, otherwise extracting the selected text result from the text result such that the selected text result (534 or 504) corresponds to the fourth coordinate in the mosaic image; Providing the selected text results (544).

1. A medical device monitoring system comprising:

2. The server, Automatically, if the selector text result is found to not match the first predefined text, comparing the selector text result to a second predefined text that is different from the first predefined text; If the selector text result is found to match the second predefined text, extracting the selected text result from the text result such that the selected text result corresponds to the fourth coordinate within the mosaic image.

2. The medical device monitoring system of claim 1, configured as follows:

3. 2. The medical device monitoring system of claim 1, wherein the at least one of the additional defined target regions in the mosaic image corresponds to an additional source region including a version number (220, 422) of software executed by the medical device control device.

4. The medical device monitoring system of claim 1 , wherein the at least one of the additional defined regions of interest in the mosaic image corresponds to an additional source region that includes an identifier of a medical device coupled to the medical device control device.

5. The medical device monitoring system of claim 4 , wherein the identifier of the medical device comprises a serial number (204, 404).

6. The medical device monitoring system of claim 4 , wherein the identifier of the medical device includes a type code (202, 402).

7. 2. The medical device monitoring system of claim 1, wherein the at least one of the additional defined destination regions in the mosaic image corresponds to an additional source region that includes an identifier of an optional feature installed on the medical device control device.

8. the defined destination region at the third coordinate in the mosaic image is larger than the first source region; wherein the server is configured to expand the first source area to fill the defined destination area at the third coordinates to copy the first source area into the defined destination area at the third coordinates.

2. The medical device monitoring system of claim 1.

9. 2. The medical device monitoring system of claim 1, wherein the server is configured to convert (720) each colored pixel of the first source region to a white pixel in the defined destination region at the third coordinate to copy the first source region to the defined destination region at the third coordinate.

10. A method for monitoring a medical device, comprising the steps of: requesting and receiving, via a computer network, from each medical device controller of a plurality of medical device controllers, the plurality of medical device controllers including a first medical device controller each configured to display a first type of information at a first screen coordinate and a second medical device controller each configured to display the first type of information at a second screen coordinate different from the first screen coordinate, a received image of content displayed on a screen of the medical device controller (800); generating a mosaic image for each received image, (a) copying (802) a first source region of the received image encompassing the first screen coordinate to a predefined destination region at a third coordinate in the mosaic image; (b) copying (806) a second source region of the received image encompassing the second screen coordinate to a predefined destination region in the mosaic image at a fourth coordinate different from the third coordinate; (c) copying (808) a plurality of additional source regions of the received image to respective additional defined destination regions in the mosaic image; A process comprising: After generating the mosaic image, subjecting the mosaic image to optical character recognition in a single service call to generate a text result (810); extracting (816) from the text results a selector text result that corresponds to at least one of the additional defined regions of interest within the mosaic image; comparing said selector text result with a first predefined text; if the selector text result is found to match the first predefined text, extracting a selected text result from the text result such that the selected text result corresponds to the third coordinate in the mosaic image (812), and otherwise extracting the selected text result from the text result such that the selected text result corresponds to the fourth coordinate in the mosaic image (814); and Providing the selected text results.

11. Otherwise, extracting the selected text result comprises: comparing the selector text result to a second predefined text different from the first predefined text; if the selector text result is found to match the second predefined text, extracting the selected text result from the text result such that the selected text result corresponds to the fourth coordinate within the mosaic image; 11. The method of claim 10, comprising:

12. 11. The method of claim 10, wherein the at least one of the additional defined regions of interest in the mosaic image corresponds to an additional source region that includes a version number of software executed by the medical device controller.

13. 11. The method of claim 10, wherein the at least one of the additional defined regions of interest in the mosaic image corresponds to an additional source region that includes an identifier of a medical device coupled to the medical device controller.

14. 14. The method of claim 13, wherein the identifier for the medical device comprises a serial number.

15. 14. The method of claim 13, wherein the identifier of the medical device comprises a type code.

16. 11. The method of claim 10, wherein the at least one of the additional defined destination regions in the mosaic image corresponds to an additional source region that includes an identifier of an optional feature installed on the medical device control device.

17. the defined destination region at the third coordinate in the mosaic image is larger than the first source region; copying the first source region into the defined destination region at the third coordinates includes expanding the first source region to fill the defined destination region at the third coordinates.

11. The method of claim 10.

18. 11. The method of claim 10, wherein copying the first source region to the defined destination region at the third coordinates includes converting each colored pixel of the first source region to a white pixel in the defined destination region at the third coordinates.

19. instructions that, when executed by a processor, establish a process for performing a computer-implemented method for monitoring a medical device.

11. A non-transitory computer readable medium encoded with a process comprising: a process configured to request and receive, via a computer network, from each medical device controller of a plurality of medical device controllers, the medical device controllers including a plurality of first medical device controllers each configured to display a first type of information at a first screen coordinate, and a plurality of second medical device controllers each configured to display the first type of information at a second screen coordinate different from the first screen coordinate, a received image of content displayed on a screen of the medical device controller; A process configured to generate a mosaic image for each received image, comprising: (a) copying a first source region of the received image encompassing the first screen coordinate to a predefined destination region at a third coordinate in the mosaic image; (b) copying a second source region of the received image encompassing the second screen coordinate to a predefined destination region in the mosaic image at a fourth coordinate different from the third coordinate; (c) copying a plurality of additional source regions of the received image to respective additional defined destination regions within the mosaic image. A process configured to: a process configured to, after generating the mosaic image, subject the mosaic image to optical character recognition in a single service call to generate a text result; a process configured to extract from the text results a selector text result that corresponds to at least one of the additional defined regions of interest in the mosaic image; a process configured to compare the selector text result with a first predefined text; a process configured to, if the selector text result is found to match the first predefined text, extract a selected text result from the text result such that the selected text result corresponds to the third coordinate in the mosaic image, and otherwise extract the selected text result from the text result such that the selected text result corresponds to the fourth coordinate in the mosaic image; A process configured to provide the selected text results.

20. If not, the process configured to extract the selected text result comprises: comparing the selector text result to a second predefined text different from the first predefined text; If the selector text result is found to match the second predefined text, extracting the selected text result from the text result such that the selected text result corresponds to the fourth coordinate within the mosaic image.

20. The non-transitory computer-readable medium of claim 19, configured to:

Images