Intelligent image segmentation before Optical Character Recognition (OCR)

Abstract

This invention provides a medical device monitoring system for remotely monitoring medical devices and a method for monitoring medical devices. [Solution] The method for monitoring medical devices extracts information from screen images from medical device control units with one OCR process call per screen image, depending on which medical device control unit's screen image is being processed, even though important information appears in different screen locations. For example, different software versions of medical device control units may display the same type of information in different screen locations. Copies of the important screen information (one copy from each different screen location) are created in a mosaic image, and the mosaic image is then OCR processed to generate text results. The text is selectively extracted from the OCR text results according to the contents of selector fields on the screen image, such as the software version number and the cardiac pump model identifier.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Patent Application No. 16 / 360,270, filed on March 21, 2019, entitled "Intelligent Image Segmentation Before Optical Character Recognition (OCR)", the entire content of which is incorporated herein by reference for all purposes.

[0002] Technical Field The present invention relates to remotely monitoring medical devices, and more particularly to intelligently segmenting an image of a screen displayed by a medical device control device before the image is subjected to optical character recognition (OCR) processing.

Background Art

[0003] Related Technologies Many medical devices, such as some intravascular blood pumps, e.g., the Impella® 2.5 heart pump available from Abiomed, Inc. of Danvers, Massachusetts, are connected to an external medical device control device that collects and displays operating data regarding the medical device, such as heart signal levels, battery temperature, blood flow rate, and tubing integrity. An exemplary medical device control device is available from Abiomed, Inc. under the trade name Automated Impella Controller®. These medical device control devices issue an alarm when an operating data value exceeds a predetermined value or range, e.g., when a leak or loss of suction is detected. These medical device control devices include a video display screen as a human interface, on which the operating data and / or an alarm are displayed.

[0004] To facilitate remote monitoring by healthcare professionals and ensure effectiveness and patient safety, some of the control systems for such medical devices may be coupled to a central server that can be accessed by a monitoring station, often via a computer network including a wireless segment. The monitoring station may display real-time operational data and / or alarms on a display screen for healthcare professionals to view.

[0005] The server requests and receives images of the content displayed on the screen of the medical device control unit. Some servers use optical character recognition (OCR) technology to parse the images and extract text information such as the heart pump serial number, blood flow rate, and warning message text. To reduce OCR errors, some servers mask portions of the screen image that contain irrelevant text or graphics that could confuse the OCR engine.

[0006] However, different medical device control systems may display the same type of information, such as blood flow rate, in different locations on their respective screens, depending, for example, on which version of software the medical device control system is running or on which type of heart pump is connected to the medical device control system. This ambiguity of screen location makes it difficult or impossible to effectively mask the appropriate portion of the screen image before OCR. Therefore, the server may use multiple masks, perform the OCR process on each mask, and then select one OCR result. However, OCR processing is expensive and time-consuming. Providers would prefer that the server perform only one OCR process per screen image. Unfortunately, the server has no predictive ability and therefore cannot decide in advance which mask to use. [Overview of the Initiative]

[0007] Overview of the characteristics One aspect of the present invention provides a medical device monitoring system. This system includes a server. The server is configured to automatically request and receive "received images". The server is configured to request and receive received images via a computer network. The server is configured to request and receive received images from each of a plurality of medical device control devices. Each received image includes content displayed on the screen of a medical device control device. The plurality of medical device control devices includes a plurality of first medical device control devices and a plurality of second medical device control devices. For example, a first medical device control device may run a specific version of software, and a second medical device control device may run a different version of software. In another example, different types of medical devices are connected to the first and second medical device control devices, respectively.

[0008] Each first medical device control unit is configured to display a first type of information at a first screen coordinate. Each second medical device control unit is configured to display a first type of information at a second screen coordinate different from the first screen coordinate. In other words, the first and second medical device control units display certain information ("first information"), such as minimum blood flow rate, at different positions on their respective screens. This difference in screen position may be due to differences in software versions or different types of medical devices connected to the first and second medical device control units.

[0009] For each received image, the server is configured to automatically generate a mosaic image. To generate the mosaic image, the server is configured to copy a first source region of the received image to a predefined target region at a third coordinate within the mosaic image. The first source region encompasses the first screen coordinate of the received image. The server is also configured to copy a second source region of the received image to a predefined target region at a fourth coordinate within the mosaic image. The fourth coordinate is different from the third coordinate. The second source region encompasses the second screen coordinate of the received image. The server is further configured to copy multiple additional source regions of the received image to their respective additional predefined target regions within the mosaic image.

[0010] After the mosaic image is generated, the server is configured to automatically perform optical character recognition on the mosaic image in a single service call to generate text results.

[0011] The server is also configured to automatically extract selector text results from the text results. The selector text results correspond to at least one of additional predefined target regions in the mosaic image. The server is configured to compare the selector text results with a first predefined text. If it is found that the selector text results match the first predefined text, the server is configured to extract the selected text results from the text results so that the selected text results correspond to a third coordinate in the mosaic image. Otherwise, the server is configured to extract the selected text results from the text results so that the selected text results correspond to a fourth coordinate in the mosaic image. The server is configured to provide the selected text results.

[0012] Therefore, while a medical device control unit may display the same information in different screen positions, the server can select the appropriate OCR result from one of the screen positions based on the content of a different field on the screen, without having to perform two separate OCR processes.

[0013] In any embodiment, the server may be configured to automatically compare the selector text result with a second predefined text if it finds that the selector text result does not match a first predefined text, the second predefined text being different from the first predefined text. If it finds that the selector text result matches the second predefined text, the server may be configured to automatically extract the selected text result from the text results such that the selected text result corresponds to a fourth coordinate in the mosaic image.

[0014] In any embodiment, at least one of the additional predefined target regions within the mosaic image may correspond to an additional source region containing the version number of the software executed by the medical device control unit.

[0015] In any embodiment, at least one of the additional predefined target regions within the mosaic image may correspond to an additional source region containing an identifier for a medical device coupled to a medical device control device.

[0016] In any embodiment, the identifier of a medical device may include a serial number.

[0017] In any embodiment, the identifier of a medical device may include a type code.

[0018] In any embodiment, at least one of the additional predefined target regions within the mosaic image may correspond to an additional source region containing an identifier for an optional function installed in a medical device control unit.

[0019] In any embodiment, a predefined target region at a third coordinate within the mosaic image may be larger than the first source region. To copy the first source region to the predefined target region at the third coordinate, the server may be configured to expand the first source region to fill the predefined target region at the third coordinate.

[0020] In any embodiment, in order to copy a first source region to a predefined target region at third coordinates, the server may be configured to convert each colored pixel in the first source region to a white pixel in the predefined target region at third coordinates.

[0021] Another aspect of the present invention provides a method for monitoring a medical device. This method includes the step of requesting and receiving a received image. The requested image is requested and received via a computer network. The received image is requested and received by each medical device control unit of a plurality of medical device control units. Each received image includes content displayed on the screen of the medical device control unit. The plurality of medical device control units includes a plurality of first medical device control units and a plurality of second medical device control units. Each first medical device control unit is configured to display a first type of information at first screen coordinates, and each second medical device control unit is configured to display a first type of information at second screen coordinates different from the first screen coordinates.

[0022] This method includes the step of generating a mosaic image for each received image. The step of generating a mosaic image includes copying a first source region containing a first screen coordinate of the received image to a predefined target region at a third coordinate in the mosaic image. A second source region containing a second screen coordinate of the received image is copied to a predefined target region in the mosaic image at a fourth coordinate different from the third coordinate. Multiple additional source regions of the received image are copied to their respective additional predefined target regions in the mosaic image.

[0023] After generating a mosaic image, the mosaic image is optically recognized in a single service call to generate text results. A selector text result is extracted from the text result. The selector text result corresponds to at least one of the additional predefined target regions in the mosaic image. The selector text result is compared to a first predefined text.

[0024] If it is found that the selector text result matches the first defined text, the selected text result is extracted from the text result such that the selected text result corresponds to the third coordinates in the mosaic image. Otherwise, the selected text result is extracted from the text result such that the selected text result corresponds to the fourth coordinates in the mosaic image. Then, the selected text result is provided.

[0025] In any aspect, in the case where it is not so, the step of extracting the selected text result may include comparing the selector text result with a second defined text. The second defined text may be different from the first defined text. If it is found that the selector text result matches the second defined text, the selected text result is extracted from the text result such that the selected text result corresponds to the fourth coordinates in the mosaic image.

[0026] In any aspect, at least one of the additional defined target regions in the mosaic image may correspond to an additional source region that includes the version number of the software executed by the medical device control device.

[0027] In any aspect, at least one of the additional defined target regions in the mosaic image may correspond to an additional source region that includes the identifier of the medical device coupled to the medical device control device.

[0028] In any aspect, the identifier of the medical device may include a serial number.

[0029] In any aspect, the identifier of the medical device may include a type code.

[0030] In any aspect, at least one of the additional defined target regions in the mosaic image may correspond to an additional source region that includes the identifier of the optional function installed in the medical device control device.

[0031] In any embodiment, the predefined target region at a third coordinate in the mosaic image may be larger than the first source region. Copying the first source region to the predefined target region at the third coordinate may include expanding the first source region to fill the predefined target region at the third coordinate.

[0032] In any embodiment, copying a first source region to a predefined target region at third coordinates may include converting each colored pixel in the first source region to a white pixel in the predefined target region at third coordinates.

[0033] A further aspect of the present invention provides a non-temporary computer-readable medium encoded with instructions. When executed by a processor, the instructions establish a process for performing a computer implementation method for monitoring a medical device. The process includes a process configured to request and receive received images of content displayed on the screens of a plurality of medical device control devices from each medical device control device via a computer network. The plurality of medical device control devices include a plurality of first medical device control devices, each configured to display a first type of information at first screen coordinates, and a plurality of second medical device control devices, each configured to display a first type of information at second screen coordinates different from the first screen coordinates.

[0034] The process also includes a process configured to generate a mosaic image for each received image. The process is configured to copy a first source region encompassing a first screen coordinate of the received image to a predefined target region at a third coordinate in the mosaic image. The process is also configured to copy a second source region encompassing a second screen coordinate of the received image to a predefined target region in the mosaic image at a fourth coordinate different from the third coordinate. The process is further configured to copy multiple additional source regions of the received image to their respective additional predefined target regions in the mosaic image.

[0035] The process further includes a process configured to generate a mosaic image and then perform optical character recognition on the mosaic image in a single service call to generate text results.

[0036] The process includes a process configured to extract selector text results from text results. The selector text results correspond to at least one of additional predefined target regions in the mosaic image. The process includes a process configured to compare the selector text results with a first predefined text.

[0037] The process includes a process configured to extract the selected text result from the text result so that the selected text result corresponds to a third coordinate in the mosaic image if the selector text result is found to match a first predefined text, and otherwise extract the selected text result from the text result so that the selected text result corresponds to a fourth coordinate in the mosaic image.

[0038] The process also includes a process configured to provide selected text results. [Brief explanation of the drawing]

[0039] The present invention will be better understood by referring to the following detailed description of specific embodiments in conjunction with the drawings.

[0040] [Figure 1] This is a perspective view of an exemplary conventional medical device control device and an exemplary conventional medical device coupled to the medical device control device, in this example a cardiac pump, based on prior art. [Figure 2] This figure shows an example of a virtual display screen content that can be displayed on the screen of the medical device control device shown in Figure 1, based on prior art. [Figure 3] This figure schematically illustrates a process for processing screen display images, such as the contents of the virtual display screen shown in Figure 2, according to one aspect of the present invention. [Figure 4]This figure shows another exemplary virtual display screen content that can be displayed on the screen of a medical device control device that runs a different version of software than the medical device control device that generated the screen content shown in Figure 2, according to prior art. [Figure 5] This figure schematically illustrates a process for processing screen display images, such as the virtual display screen content of Figure 2 and the virtual display screen content of Figure 4, according to one aspect of the present invention, regardless of the differences between the virtual display screen content of Figure 2 and the virtual display screen content of Figure 4. [Figure 6] This is a schematic block diagram of the main components of a medical device monitoring system for collecting, storing, and acquiring operational data relating to multiple medical device control devices from multiple medical device control devices, including a server that performs the processing steps shown in Figure 5, according to one aspect of the present invention. [Figure 7] This is a schematic block diagram of the server shown in Figure 6, according to one aspect of the present invention. [Figure 8] This flowchart schematically illustrates a method for monitoring medical devices, such as the method performed by the servers shown in Figures 6 and 7, according to one aspect of the present invention. [Figure 9] This flowchart schematically illustrates the details of the operation of the flowchart in Figure 8 (selecting one text result based on additional text results) according to one aspect of the present invention. [Figure 10] This flowchart schematically illustrates the operation performed by an arbitrary color pixel whitener of the server shown in Figures 6 and 7, according to one aspect of the present invention. [Modes for carrying out the invention]

[0041] Detailed description of a specific aspect Aspects of the present invention provide a medical device monitoring system and a medical device monitoring method that extract information from screen images from medical device control devices with a single OCR process call for each screen image, even though important information appears in different positions on the screen depending on which medical device control device screen image is being processed.

[0042] Figure 1 is a perspective view of an exemplary conventional medical device control unit 100 and an exemplary conventional medical device 102 coupled to the medical device control unit 100, which in this example is a heart pump. In the example shown in Figure 1, the medical device control unit 100 is an Automated Impella Controller® manufactured by Abiomed, Inc. of Danvers, Massachusetts, and the heart pump 102 is an Impella® 2.5 heart pump, also available from Abiomed, Inc., but any suitable medical device control unit may be used. In some cases, a medical device and its associated medical device control unit are combined. Such a combination is referred to herein simply as a medical device control unit.

[0043] The medical device control unit 100 includes a display screen 104 on which it displays operational data related to the medical device, such as cardiac signal levels, battery temperature, blood flow rate, and piping integrity. As will be discussed in more detail herein, the medical device control unit 100 may be connected to a computer network, thereby transmitting images of the content displayed on the screen 104 to a remote server (not shown).

[0044] Figure 2 shows an exemplary virtual display screen content 200 that may be displayed on screen 104 of the medical device control device 100 in Figure 1. For example, the display screen content may include the heart pump type ("Impella 5.0") 202, the heart pump serial number ("120703") 204, one or more warning / error / information messages 206, a placement signal 207, the current heart pump speed (performance) setting ("P-0") 208, the heart pump motor current value 210, the current blood flow rate 212, and the minimum and maximum blood flow rates 213. The display screen content 200 is typically pixelated.

[0045] Figure 3 schematically shows the process for processing screen display images 300, such as the virtual display screen content 200 in Figure 2. Although Figure 3 is a schematic diagram, the parts of Figure 3, such as the screen image 300, show the approximate location and size of specific items, such as the cardiac pump type source area 306, graph 318, and graph 320, which correspond to the exemplary virtual display screen content 200 shown in Figure 2.

[0046] As described above, to reduce OCR errors, some servers mask portions of screen images 200, 300 that contain irrelevant text or graphics that could confuse the OCR engine 302. For example, the virtual display screen content 200 (Figure 2) includes graphs 214 and 216, as well as legends such as "Impella Flow" 218, which, if included in the input to the OCR engine 302, could cause or contribute to OCR errors.

[0047] To avoid these errors, a mosaic image 304 is generated from only a predetermined portion of the screen display image 300. The mosaic image 304 may or may not have the same dimensions as the screen display image 300 in pixel units. In this example, three source regions 306, 308, and 310 of the screen display image 300 are copied to their respective predefined target regions 312, 314, and 316 of the mosaic image 304. However, any number of source regions of the screen display image 300 may be copied to the mosaic image 304.

[0048] Each source region 306–310 can be identified by its corresponding screen coordinates. As used herein, the terms screen coordinates, or simply coordinates, mean information that identifies the location and size of an area of ​​an image, i.e., the entire space in which information is displayed or placed within the image. An area may or may not be rectangular. The coordinates may include, for example, the x and y pixel positions of the lower-left corner of the area, as well as the height and width of the area in pixel units. Other appropriate units of measurement and other appropriate representations may be used, as long as they identify the location and extent of the area. Thus, fields of different widths will have different coordinates, even if both fields start at the same x and y positions.

[0049] The target regions 312-316 may or may not have the same coordinates as their respective corresponding source regions 306-310. That is, the target regions 312-316 may or may not have the same starting position and / or size as their respective corresponding source regions 306-310. In some embodiments, one or more of the target regions 312-316 are larger than their corresponding source regions 306-310.

[0050] In this case, the server enlarges the source area 312-316 so that the enlarged result fills the corresponding target area 312-316. Exemplary magnifications include, but are not limited to, 1.1x, 1.5x, 2x, and 3x. Such magnifications may reduce the possibility of OCR errors.

[0051] In some embodiments, the server changes the color of pixels when they are copied from source regions 306-310 to target regions 312-316. For example, the server may convert colored pixels to white pixels to increase contrast and thus reduce OCR errors. The server may, for example, convert pixels to white pixels whose total brightness, determined by the sum of the red, blue, and green values ​​of the pixel, exceeds a threshold, and the server may convert all remaining pixels to black.

[0052] Irrelevant portions of the screen display image 300, such as graphs 318 and 320 (corresponding to graphs 214 and 216 in Figure 2), are not copied to the mosaic image 304. Screen coordinates of source regions 306-310 should be selected to avoid text and / or graphics that, if included, could confuse the OCR engine 302 or cause the OCR engine 302 to include false text in its output. The rest of the mosaic image 304 remains blank.

[0053] Once the mosaic image 304 is generated, it is passed to the OCR engine 302 for conversion to text. The server may include the OCR engine 304, or it may call an external OCR engine, such as the OCR engine of the Google Cloud Vision API provided by Google (Alphabet, Inc., Mountain View, California). The OCR engine 302 returns a text result 321 represented by text 1 322, text 2 324, and text 3 326, corresponding to the target regions 312-316, respectively. The server extracts text 1 322, text 2 324, and text 3 326 from the text result 321.

[0054] Optionally, text result 321, or parts thereof, such as text 1, text 2, and / or text 3 of 322-326, may be validated using predefined rules such as "the pump serial number must be a 5-7 digit number." Such validation can prevent some OCR errors that would corrupt subsequent displays, etc. Optionally, text result 321, or parts thereof 322-326, may be tidied up according to predefined procedures such as "remove leading and trailing spaces from error messages." These predefined rules and procedures may be stored in a configuration file, which will be described in more detail herein.

[0055] As described above, different medical device control units 100 (Figure 1) may display the same type of information, such as blood flow rate, in different locations on their respective screens 104, depending, for example, which version of software the medical device control unit is running, or which type of heart pump 102 is connected to the medical device control unit 100. Figure 4 shows another exemplary virtual display screen content 400 that may be displayed on the screen 104 of a different medical device control unit 100, for example, one that is running a different version of software than the medical device control unit that generated screen content 200 (Figure 2), or one that is connected to a different type of heart pump 102, such as an Impella routine CP heart pump from Abiomed, Inc.

[0056] In the exemplary virtual display screen content 400, the cardiac pump type ("Impella CP") 402 differs from the corresponding cardiac pump type 202 shown in screen content 200 (Figure 2), indicating that a different type of cardiac pump is connected to the medical device control unit that generated the display screen content 400. The cardiac pump serial number is shown in 404. As can be seen by comparing Figure 4 and Figure 2, some of the information in display content 400 is displayed in different locations than the corresponding information in display content 200. For example, the placement signal 407, motor current 410, and current, minimum, and maximum blood flow rates 412 and 413 are, respectively, at screen coordinates (position and / or size) from the same information 207, 210, 212, and 213 in screen content 200 (Figure 2).

[0057] Clearly, this difference in screen coordinates causes problems for a single server that receives screen images from both medical device control units and attempts to mask the relevant parts of the screen in the manner discussed with respect to Figure 3. Simply defining a source area large enough to encompass both versions of the display would fail, as such a large source area, e.g., source area 420, would contain irrelevant text ("(mmHg)") within display content 400.

[0058] An aspect of the present invention solves this problem, as schematically shown in Figure 5. Although Figure 5 is a schematic diagram, parts of Figure 5, such as the screen image 300, show the approximate location and size of specific items, such as the heart pump type 306, graph 318, and graph 320, which correspond to the exemplary virtual display screen content 200 and virtual display screen content 400 shown in Figures 2 and 4. Figure 5 schematically shows the process for processing the screen display image 300, such as the virtual display screen content 200 in Figure 2 and the virtual display screen content 400 in Figure 4, regardless of the differences between the virtual display screen content 200 and the virtual display screen content 400.

[0059] Aspects of the present invention solve the problem by creating copies of each source region that appear at different screen coordinates on the respective screens 104 of different medical device control devices 100. Each of these copies is processed by an OCR engine 302 to potentially generate multiple text results, for example, text 5 502 and text 6 504. After the mosaic image 506 is processed by the OCR engine 302, the medical device control device 100 generates a screen image 300 based on it, and one of the text results, text 5 502 or text 6 504, is selected. Which medical device control device 100 generated the screen image may be determined, for example, by the software version number 508 displayed on the screen 104 and processed by the OCR engine 302, as will be described in detail below.

[0060] Continuing with the exemplary virtual display screen contents 200 and 400, as described above, the placement signal 207 (Figure 2) appears at different screen coordinates in display screen content 200 than the placement signal 407 (Figure 4) in display screen content 400. As shown in Figure 5, the placement signal 207 (from Figure 2) appears in source region 510, while the placement signal 407 (from Figure 4) appears in source region 512. Source region 512 overlaps with source region 512. Each of these source regions 510 and source region 512 is copied to separate predefined target regions 514 and 516 in the mosaic image 506, respectively. Ideally, target regions 514 and 516 do not overlap.

[0061] Similarly, the current blood flow rate 212 (Figure 2) appears in a different screen coordinate within the display screen content 200 than the current blood flow rate 412 (Figure 4) within the display screen content 400. As shown in Figure 5, the current blood flow rate 212 (from Figure 2) appears in source region 310, as seen in Figure 3, while the current blood flow rate 412 (from Figure 4) appears in source region 518. Each of these source regions 310 and 518 is copied to separate target regions 520 and 522, respectively, within the mosaic image 506. Ideally, target regions 520 and 522 do not overlap.

[0062] Source region 310 overlaps with source region 518. However, the overlap of the two source regions is unnecessary. Any other text and / or graphics adjacent to one or both of source regions 310 and / or 518 can be avoided by copying source regions 310 and 518 to separate target regions 520 and 522.

[0063] Source regions that do not cause problems for the OCR engine 302 are each copied to only one target region. For example, if heart pump type 202 (Figure 2) occupies the same screen coordinates as heart pump type 404 (Figure 4), the source region containing the text of the heart pump type on screen 104 is copied to a single target region 526 in the mosaic image 506. Even if heart pump type 202 (Figure 2) does not occupy the same screen coordinates as heart pump type 404 (Figure 4), but a single larger source region can contain both heart pump type fields 202 and 404 without the risk of OCR errors, the larger source region can still be copied to a single target region.

[0064] Similarly, if the software version number occupies the same screen coordinates in both virtual display screen contents 200 and virtual display screen contents 400, or if a larger source area can encompass both software version numbers without the risk of OCR errors, the source area 508 can be copied into a single target area 528.

[0065] The coordinates of various source regions 310, 508, 510, 512, 518, and 524, as well as the coordinates of various target regions 514, 516, 520, 522, 526, and 528, may be stored in a configuration file rather than hardcoded, for example, to facilitate the addition and adjustment of these values ​​in response to changes in the software performed by the medical device control device 100.

[0066] Once the mosaic image 504 is generated, it is passed to the OCR engine 302 for conversion to text. The OCR engine 302 returns a text result 530 represented by text 1 532, text 2 534, text 3 536, text 4 538, text 5 502, and text 6 504, corresponding to the target regions 514, 516, 528, 526, 520, and 522, respectively, as discussed herein with respect to Figure 3. The server extracts text 1 532, text 2 534, ..., text 6 504 from the text result 530.

[0067] Assuming that the difference between the virtual display screen content 200 and the virtual display screen content 400 is due to the software versions running on the respective medical device control devices 100 that generated the screen content 200 and screen content 400, the server extracts text 3 536 from the text result 530. Text 3 536 includes optical character recognition of the software version number displayed on screen 104, for example, as shown in 220 (Figure 2). The selector 540 selects either text 5 502 or text 6 504 depending on the value of text 3 536, i.e., the software version. For example, the selector 540 may compare text 3 536 with possible values ​​of the software version and then select either text 5 502 or text 6 504 depending on which of the possible values ​​of the software version matches the actual software version value text 3 536. Thus, text 3 536 functions as input 542 to the selector 540. The text selected by the selector 540 is provided as output 544 from the selector 540.

[0068] The selected text 544 may then be processed by another component, such as a comparator, which compares the value to predetermined minimum and maximum values ​​and issues an alarm if the current value is outside the range defined by the minimum and maximum values.

[0069] Similarly, the selector 540 selects from text 1 532 and text 2 534, corresponding to target area 514 and target area 516, respectively. Although the example in Figure 5 illustrates selection from only two options, three or more types of medical device control devices 100 may transmit screen images 300. In this case, each of several of the source areas 310, 508-512, 518, and 524 may be copied to three, four, or more separate target areas within the mosaic image 506, as necessary to avoid ambiguity in optical character recognition.

[0070] In the example described above, the version number of the software running on the medical device control unit 100 is used to select from the target areas 520, 522 and 514, 516 to be duplicated. However, in other cases, other information that can be extracted from the display image 300 may be used to provide input 542 to the selector 540. For example, the cardiac pump type 202 (Figure 2), which is included by the source area 524 and target area 526 and corresponds to the text 4 538, can be used as input 542 to the selector 540, where appropriate.

[0071] Preferably, the text or other indicators from the display image 300 used to derive the input 542 to the selector 540 reside at the same screen coordinates on each screen 104 of different medical device control devices 100. However, as with the software version number, the text or other indicators may be located at different coordinates on the screen 104 of different medical device control devices 100, as illustrated by the virtual display screen content 200 (Figure 2) and the virtual display screen content 400 (Figure 4). In some cases, the source area used to drive the selector 540, for example, the source area 508, does not have to encompass all possible screen positions on which the text or other indicators may be displayed, as long as the selector 540 receives enough input 542 to make a selection.

[0072] For example, if source area 508 contains software version 220 (Figure 2) in virtual display screen content 200 but does not contain software version 422 (Figure 4) in virtual display screen content 400, and text 3 536 contains "X7.1" (the software version number displayed in virtual display screen content 200), then selector 540 may select text 5 502. If text 3 536 does not contain "X7.1", then selector 540 may select text 6 504. In other words, if input 542 does not match any predetermined value, selector 540 will make a default predetermined selection such as text 3 536.

[0073] In this example, only the objective domain 528 is used to drive the selector 540, but the input 542 to the selector 540 may contain text from any combination of objective domains 514, 516, 520, 522, 526, and 528. For example, if the software version number is greater than a certain value and the heart pump is a specific model, an OCR value derived from one objective domain may be used, but if the software version number is less than or equal to a certain value, an OCR value derived from a different objective domain may be used. Furthermore, the decision process performed by the selector 540 may include several steps. For example, if the medical device is a specific heart pump model, the software version number may be used to make the selection, but if the medical device is a different specific heart pump model, the heart pump serial number may be used to make the selection.

[0074] Examples of text or other indicators from display image 300 that may be used to derive input 542 to selector 540 include the software version, hardware version, heart pump model identifier, heart pump serial number, display of optional features or components present in or installed on the heart pump or medical device control unit, display of options selected by the user or administrator (for example, to customize how the information is displayed on screen 104), and combinations thereof.

[0075] Figure 6 is a schematic block diagram of the main components of a medical device monitoring system 600 for collecting, storing, and acquiring operational data from multiple medical device control units 100. For simplicity, only medical device control units 100 are shown in Figure 6, and individual medical devices are not shown. Three medical device control units 100 are shown, but other numbers of medical device control units 100 may be used. Each medical device control unit 100 is optionally connectable to a computer network 602 via a remote link module 604. Each medical device control unit 100 is configured to automatically and repeatedly retrieve status information about the connected medical device and display the status information on a display screen 104 (Figure 1). As described above, Figures 2 and 4 show the respective virtual display screen contents 200 and 400 that may be displayed on the screen 104 of any given medical device control unit 100.

[0076] Server 606 is configured to automatically request and receive images of the content displayed on the screen 104 of each medical device control unit 100, either periodically or as needed, usually every 20 seconds. The requests and images are transmitted via the computer network 602. The images may be transmitted as one or more messages encoded as video frames or sequences of video frames. A video frame may, for example, include a pixelated copy of an image displayed on the display screen 104 of the medical device control unit 100.

[0077] Server 606 is configured to process received frames (images) as described with respect to Figures 2 to 5. As described above, verification rules, sorting procedures, and coordinates of various source regions 310, 508, 510, 512, 518 and 524, as well as coordinates of various target regions 514, 516, 520, 522, 526 and 528, may be stored in configuration file 607. Server 606 may communicate with an external OCR engine (not shown) via computer network 602. Server 606 can then automatically verify, using recognized and possibly selected text, the serial number or other identifier of the medical device control unit 100, the operating parameters of the medical device control unit 100, whether an alarm has been issued by one of the medical device control units 100, etc.

[0078] The data store 608 is configured to store one or more media files, particularly frames (images) such as MP4 videos or other suitable types of media files, and the server 606 is configured to automatically store received frames (images) in the data store 608. The data store 608 records screen images received by the server 606 for later playback, such as in response to a request from one of several monitoring stations 610. The data store 608 is configured to provide the requested portion of the stored media files in response to a provision request. The data store 608 thus supports the playback of status information of the medical device control device 100. For example, the data store 608 may provide one or more frames (images) of a video stored in a media file for display to a user.

[0079] The server 606 may also be configured to provide status information regarding one or more of the medical device control devices 100 to multiple of the monitoring stations 610 based on images received in real time by the server 606 and / or based on historical information held in the data store 608.

[0080] Figure 7 is a schematic block diagram of the server 606 in Figure 6. The server 606 includes a mosaic image generator 700 configured to produce a mosaic image 506 (Figure 5). The mosaic image generator 700 includes a first source area copy 704, a second source area copy 706, and an additional source area copy 708. The first, second, and additional source area copies 704-708 are shown as separate components, but some or all of these copies 704-708 may share code or other components.

[0081] For each piece of information that appears at different screen coordinates within the virtual display screen content 200 and the virtual display screen content 400 and which could cause an OCR error if extracted as a single source region from the screen image 300 (Figure 5), the first source region copyper is configured, as discussed herein, to copy one of the source regions contained within the first screen coordinates in the screen image 300 to a predefined target region at a third coordinate in the mosaic image 506. Similarly, the second source region copyper is configured, as discussed herein, to copy the other source region contained within the second screen coordinates in the screen image 300 to a predefined target region at a fourth coordinate in the mosaic image 506 that is different from the third coordinate.

[0082] Examples of such source regions include current blood flow source regions 310 and 518, which are copied to target regions 520 and 522, respectively, and placement signal source regions 510 and 512, which are copied to target regions 514 and 516, respectively.

[0083] The additional source region copier 708 is configured to copy source regions of the screen image 300 that should not cause OCR errors into the mosaic image 506, as discussed herein. The additional source region copier 708 is configured to copy multiple additional source regions of the screen image 300 into their respective additional predefined target regions within the mosaic image 506. Examples of such source regions include the pump-type source region 524 and the software version number source region 508, which are copied into target regions 526 and 528, respectively.

[0084] As described above, server 606 may include an OCR engine, or server 606 may make service calls to an external OCR engine. Both of these embodiments are represented by the OCR engine 302 in Figures 5 and 7. After the mosaic image 506 is generated, the OCR engine parses the mosaic image 506 and returns the text result 530.

[0085] The text extractor 710 is configured to extract a first text result 712 from the text result 530. The first text result 712 corresponds to a third coordinate in the mosaic image 506. For example, the first text result 712 may correspond to the current blood flow source region 310 and the corresponding target region 520.

[0086] The text extractor 710 is also configured to extract a second text result 714 from the text result 530. The second text result 714 corresponds to a fourth coordinate in the mosaic image 506. For example, the second text result 714 may correspond to the current blood flow source region 518 and the corresponding target region 522.

[0087] The text extractor 710 is further configured to extract at least one additional text result 716 from the text result 530. The at least one additional text result 716 corresponds to at least one additional predefined target region in the mosaic image 506. For example, the at least one additional text result 716 may correspond to a pump-type source region 524 corresponding to a target region 526, and / or the at least one additional text result 716 may correspond to a software version number source region 508 corresponding to a target region 528.

[0088] The text selector 540 selects either the first text result 712 or the second text result 714 based on the additional text result 716. For example, the text selector 540 may select either the first text result 712 or the second text result 714 based on the software version or the type of heart pump. The text selector 540 provides the selected text 544.

[0089] As described above, server 606 can expand a specific source region. Optionally, source region expander 718 may be configured to expand the first source region to fill a predefined target region at a third coordinate.

[0090] As described above, the server 606 can convert colored pixels to white pixels. Optionally, the color pixel whitener 720 may be configured to convert each colored pixel in the first source region to a white pixel in a predefined target region at a third coordinate.

[0091] Figure 8 is a schematic flowchart illustrating the operation of a method for monitoring a medical device. This method may be performed by a server 606 (Figures 6 and 7), and the server 606 may be configured to perform this method. In operation 800, the method includes the step of requesting and receiving an image ("received image"). The received image is requested and received via a computer network. The received image is requested and received by each medical device control unit of a plurality of medical device control units. Each received image includes content displayed on the screen of the medical device control unit. The plurality of medical device control units includes a plurality of first medical device control units, each configured to display a first type of information at first screen coordinates, and a plurality of second medical device control units, each configured to display a first type of information at second screen coordinates different from the first screen coordinates.

[0092] In 802, a mosaic image is generated for each received image. The process of generating each mosaic image includes copying a first source region encompassing the first screen coordinates of the received image (804) to a predefined target region at a third coordinate in the mosaic image; copying a second source region encompassing the second screen coordinates of the received image (806) to a predefined target region at a fourth coordinate different from the third coordinate in the mosaic image; and copying multiple additional source regions of the received image (808) to their respective additional predefined target regions in the mosaic image.

[0093] After generating the mosaic image, at 810, the mosaic image is optically recognized in a single service call to generate text results.

[0094] At step 812, a first text result is extracted from the text result. The first text result corresponds to the third coordinate in the mosaic image. At step 814, a second text result is extracted from the text result. The second text result corresponds to the fourth coordinate in the mosaic image. At step 816, at least one additional text result is extracted from the text result. At least one additional text result corresponds to at least one of the additional predefined target regions in the mosaic image.

[0095] In step 818, one of the first text result and the second text result is selected based on at least one additional text result.

[0096] Figure 9 is a schematic flowchart illustrating operation 818 (Figure 8), which selects one of the first and second text results based on at least one additional text result, assuming the selection is based on a version number, with possible version numbers being "V1", "V2", etc. Naturally, if the selection criteria are different, the flowchart may be modified accordingly. At 900, the additional text 716 (Figure 7) is compared to "V1". If the additional text 716 contains (or is equal to) "V1", control proceeds to 902, where the first text result 712 from the OCR engine 302 is selected. Control then proceeds to 904, where the selected text is returned as text result 544.

[0097] On the other hand, if in operation 900 at least one additional text result does not contain (or is not equal to) "V1", control proceeds to 906, where the additional text 716 is compared to "V2". If the additional text 716 contains (or is equal to) "V2", control proceeds to 908, where a second text result 714 from the OCR engine 302 is selected. Control then proceeds to 904, where the selected text is returned as text result 544.

[0098] As indicated by the ellipsis 910, additional checks may be performed to compare the additional text 716 with other predetermined values. However, if no match is found in any of the comparisons 900, 906, etc., control proceeds to 912 to discard the received image 300 (Figure 5).

[0099] Figure 10 is a flowchart illustrating the operations performed by an arbitrary color pixel whitener 720 (Figure 7). At 1000, the loop begins. The loop is traversed once for each pixel (referred to as the "current pixel"). At 1002, if the current pixel is not black (or if the sum of the pixel's red, green, and blue values ​​exceeds a predetermined threshold), control proceeds to 1004, where the pixel's red value is set to its maximum possible value, such as FF (hexadecimal). From there, control proceeds consecutively to 1006 and 1008, where the pixel's green value is set to its maximum possible value and the pixel's blue value is set to its maximum possible value, respectively. Control then proceeds to 1010.

[0100] On the other hand, if the current pixel is black at 1002 (or if the sum of the pixel's red, green, and blue values ​​is below a predetermined threshold), control proceeds to 1010, where it is determined whether there are still pixels in the image to be processed. If there is at least one more pixel to be processed, control proceeds to 1012, where the current pixel is advanced to the next pixel, and then the loop returns control to 1002. On the other hand, if all pixels have been processed, control proceeds from 1010 to 1014.

[0101] While the present invention has been described through the exemplary embodiments described above, modifications and variations thereof can be made to the exemplary embodiments without departing from the inventive concept disclosed herein. For example, certain parameter values ​​such as display screen field names (e.g., cardiac pump type, placement signal, blood flow rate, software version number, and cardiac pump serial number) may be referenced in relation to the embodiments of disclosure, but within the scope of the invention, the values ​​of all parameters can vary widely to suit different applications. For example, fields other than the software version number and / or cardiac pump type may be used to distinguish screen images generated by different medical device control devices.

[0102] As used herein, including in the attached claims, the term "and / or" as used in relation to a list of items means one or more items in the list, i.e., at least one of the items in the list, but not necessarily all of the items in the list. The term "or" as used herein, including in the attached claims, as used in relation to a list of items means one or more items in the list, i.e., at least one of the items in the list, but not necessarily all of the items in the list. "Or" does not mean "exclusive OR".

[0103] While aspects of the embodiments may be described in relation to flowcharts and / or block diagrams, the functions, operations, decisions, etc., of all or part of each block, or combination of blocks, may be combined, separated into separate operations, or performed in other orders. References to “modules,” “operations,” “processes,” and similar terms are for convenience only and are not intended to limit their embodiments. All or part of each block, module, operation, process, or combination thereof may be implemented as computer program instructions (such as software), hardware (combinatorial logic, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), processors, or other hardware), firmware, or a combination thereof.

[0104] The server 606 or a portion thereof may be implemented by one or more processors that execute instructions stored in memory or are controlled by instructions stored in memory. Each processor may be a general-purpose processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated processor, or a combination thereof, as required.

[0105] The memory may be random-access memory (RAM), read-only memory (ROM), flash memory, or any other memory, or a combination thereof, suitable for storing control software or other instructions and data. Instructions defining the function of the present invention may be sent to the processor in many forms, including but not limited to information permanently stored on tangible non-temporary, non-writable storage media (e.g., read-only memory devices in a computer such as ROM, or devices readable by computer I / O attachments such as CD-ROMs and DVD discs), information modifiable on tangible non-temporary, writable storage media (e.g., floppy disks, removable flash memory, and hard drives), or information transmitted to the computer via a communication medium, including wired or wireless computer networks. Furthermore, while embodiments may be described in relation to various exemplary data structures, the system may be embodied using a variety of data structures.

[0106] Aspects or parts of the disclosure may be combined in ways not described above and / or expressly claimed. In addition, the embodiments disclosed herein can be adequately implemented without any elements not specifically disclosed herein. Therefore, the present invention should not be considered limited to the embodiments disclosed.

[0107] Where used herein, numerical terms such as “first,” “second,” and “third” are used to distinguish between the respective source and target areas and are not intended to indicate any particular order or total number of source or target areas in any particular embodiment. For example, a given embodiment may include only the second and third target areas.

Claims

1. A medical device monitoring system (600) including a server (606), wherein the server is Automatically, From each medical device control unit (100) of a plurality of medical device control units, including a plurality of first medical device control units each configured to display a first type of information (207 or 212) at first screen coordinates, and a plurality of second medical device control units each configured to display the first type of information (407 or 412) at second screen coordinates different from the first screen coordinates, via a computer network (602), a received image (300) of the content displayed on the screen (104) of the medical device control unit is requested and received (800). A mosaic image (506) is generated for each received image (802), and here the server, (a) Copy the first source region (510, 310) of the received image that includes the first screen coordinates to the defined target region (514, 520) at the third coordinates in the mosaic image (804), (b) Copy the second source region (512, 518) of the received image that includes the second screen coordinates to a predefined target region (516, 522) in the mosaic image that is at a fourth coordinate different from the third coordinate (806), (c) Copy the multiple additional source regions (508, 524) of the received image to their respective additional predefined target regions (528, 526) in the mosaic image (808) It is configured in such a way, After the mosaic image is generated, the mosaic image is subjected to optical character recognition (810, 302) in a single service call in order to generate text results (530). From the text results, extract the selector text result (536 or 538) corresponding to at least one of the additional predefined target regions in the mosaic image. The selector text result is compared with the first predefined text, If it is found that the selector text result matches the first predefined text, the selected text result is extracted from the text results such that the selected text (532 or 502) result corresponds to the third coordinate in the mosaic image; otherwise, the selected text result is extracted from the text results such that the selected text result (534 or 504) corresponds to the fourth coordinate in the mosaic image. The selected text result (544) is provided. A medical device monitoring system configured in such a way.

2. The aforementioned server, Automatically, If it is found that the selector text result does not match the first predefined text, the selector text result is compared with a second predefined text that is different from the first predefined text. If it is found that the selector text result matches the second predefined text, the selected text result is extracted from the text results such that the selected text result corresponds to the fourth coordinate in the mosaic image. A medical device monitoring system according to claim 1, configured as follows.

3. The medical device monitoring system according to claim 1, wherein at least one of the additional predefined target regions in the mosaic image corresponds to an additional source region containing the version numbers (220, 422) of the software executed by the medical device control device.

4. The medical device monitoring system according to claim 1, wherein at least one of the additional predefined target regions in the mosaic image corresponds to an additional source region containing an identifier for a medical device coupled to the medical device control device.

5. The medical device monitoring system according to claim 4, wherein the identifier of the medical device includes a serial number (204, 404).

6. The medical device monitoring system according to claim 4, wherein the identifier of the medical device includes a type code (202, 402).

7. The medical device monitoring system according to claim 1, wherein at least one of the additional predefined target regions in the mosaic image corresponds to an additional source region containing an identifier for an optional function installed in the medical device control device.

8. The defined target region at the third coordinate within the mosaic image is larger than the first source region. In order to copy the first source region to the defined target region at the third coordinates, the server is configured to expand the first source region to fill the defined target region at the third coordinates. A medical device monitoring system according to claim 1.

9. The medical device monitoring system according to claim 1, wherein the server is configured to convert each colored pixel in the first source region to a white pixel in the predefined target region at the third coordinates (720) in order to copy the first source region to the predefined target region at the third coordinates.

10. A method for monitoring medical devices, including the following steps: A process (800) of requesting and receiving received images of the content displayed on the screen of a medical device control device from each medical device control device of a plurality of medical device control devices, which includes a plurality of first medical device control devices each configured to display a first type of information at first screen coordinates, and a plurality of second medical device control devices each configured to display the first type of information at second screen coordinates different from the first screen coordinates, via a computer network; A process for generating a mosaic image for each received image, (a) Copying the first source region encompassing the first screen coordinates of the received image to a predefined target region at the third coordinates in the mosaic image (802), (b) Copying the second source region encompassing the second screen coordinates of the received image to a predefined target region in the mosaic image that is at a fourth coordinate different from the third coordinate (806), (c) Copying a plurality of additional source regions of the received image to each of the additional predefined target regions in the mosaic image (808) Processes including; After generating the aforementioned mosaic image, the process (810) involves performing optical character recognition on the mosaic image in a single service call to generate text results; Step (816): Extracting selector text results from the text results that correspond to at least one of the additional predefined target regions in the mosaic image; A step of comparing the selector text result with a first predefined text; If it is found that the selector text result matches the first predefined text, the selected text result is extracted from the text result so that the selected text result corresponds to the third coordinate in the mosaic image (812); otherwise, the selected text result is extracted from the text result so that the selected text result corresponds to the fourth coordinate in the mosaic image (814); and A step of providing the selected text result.

11. Otherwise, the step of extracting the selected text results is performed. The selector text result is compared with a second predefined text that is different from the first predefined text, If it is found that the selector text result matches the second predefined text, the selected text result is extracted from the text result such that the selected text result corresponds to the fourth coordinate in the mosaic image. The method according to claim 10, including the method described in claim 10.

12. The method according to claim 10, wherein at least one of the additional predefined target regions in the mosaic image corresponds to an additional source region containing the version number of the software executed by the medical device control device.

13. The method according to claim 10, wherein at least one of the additional predefined target regions in the mosaic image corresponds to an additional source region containing an identifier of a medical device coupled to the medical device control device.

14. The method according to claim 13, wherein the identifier of the medical device includes a serial number.

15. The method according to claim 13, wherein the identifier of the medical device includes a type code.

16. The method according to claim 10, wherein at least one of the additional predefined target regions in the mosaic image corresponds to an additional source region containing an identifier for an optional function installed in the medical device control unit.

17. The defined target region at the third coordinate within the mosaic image is larger than the first source region. Copying the first source region to the predefined target region at the third coordinates includes expanding the first source region to fill the predefined target region at the third coordinates. The method according to claim 10.

18. The method according to claim 10, wherein copying the first source region to the predefined target region at the third coordinates includes converting each colored pixel of the first source region to a white pixel in the predefined target region at the third coordinates.

19. When executed by the processor, it establishes a process for implementing a computer method to monitor medical devices, instructions A non-temporary computer-readable medium encoded in such a way that the process includes the following: A process configured to request and receive received images of the content displayed on the screen of a medical device control device from each of a plurality of medical device control devices, which includes a plurality of first medical device control devices each configured to display a first type of information at first screen coordinates, and a plurality of second medical device control devices each configured to display the first type of information at second screen coordinates different from the first screen coordinates, via a computer network. A process configured to generate a mosaic image for each received image, (a) Copy the first source region encompassing the first screen coordinates of the received image to a predefined target region at the third coordinates in the mosaic image, (b) Copy the second source region encompassing the second screen coordinates of the received image to a predefined target region in the mosaic image that is at a fourth coordinate different from the third coordinate, (c) Copy the multiple additional source regions of the received image to their respective additional predefined target regions in the mosaic image. The process is structured in such a way, A process configured to perform optical character recognition on the mosaic image in a single service call in order to generate text results after generating the aforementioned mosaic image, A process configured to extract selector text results corresponding to at least one of the additional predefined target regions within the mosaic image from the text results, A process configured to compare the selector text result with a first predefined text, A process configured such that, if the selector text result is found to match the first predefined text, the selected text result is extracted from the text result so that the selected text result corresponds to the third coordinate in the mosaic image; otherwise, the selected text result is extracted from the text result so that the selected text result corresponds to the fourth coordinate in the mosaic image. A process configured to provide the selected text result.

20. Otherwise, the process configured to extract the selected text results, The selector text result is compared with a second predefined text that is different from the first predefined text. If it is found that the selector text result matches the second predefined text, the selected text result is extracted from the text results such that the selected text result corresponds to the fourth coordinate in the mosaic image. A non-temporary computer-readable medium according to claim 19, configured as such.

Images