Computer vision for continuous data extraction and digitization from display-equipped patient care devices

Abstract

A computer vision system for continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices. A digital camera continually captures images of the patient care devices, which images are subjected to edge detection and generation of bounding rectangles around imaged objects. Image contents of each bounding rectangle are evaluated against unique class objects in a reference repository that denote classified patient care devices as candidate identities for the imaged objects. For each imaged object matched to a classified device, a device-specific pre-OCR image manipulation routine is performed in order to achieve optimal results from a subsequent optical character recognition (OCR) process that extracts text information from the manipulated image. Output data digitally embodying the text information garnered from the OCR process is communicated to recipient computing device, enabling real time monitoring and / or logging of displayed informational output from the patient care devices.

Description

[0001] COMPUTER VISION FOR CONTINUOUS DATA EXTRACTION AND DIGITIZATION FROM DISPLAY-EQUIPPED PATIENT CARE DEVICES

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority benefit of United States Provisional Patent Application No. 63 / 652,06, filed May 27, 2027, the entirety of which is incorporated herein by reference.

[0004] FIELD OF THE INVENTION

[0005] The present invention relates generally to computer vision systems, and more particularly to exploitation thereof to enable real-time monitoring of display informational output from display-equipped patient care devices.

[0006] BACKGROUND

[0007] When it comes to patient-side (e.g. bedside) patient care devices, many such devices lack the ability to export information in a digital format, which is related to not having an accessible format to export real-time continuous data (limitations of the devices themselves or being locked behind proprietary software). Thus, in many medical settings treatment information must be collected manually through individual personnel. This takes up valuable time and energy from medical professionals and limits the data collected to momentary epochs of time. Furthermore, such data transcription methods expose patients to medial errors and medical personnel to burnout. As treatment becomes personalized and continuous, the limitations present with data collection becomes a significant barrier to future medical advancement. [1 ,2] Interestingly, the information pertinent to data collection is attainable on the display screen of the patient care device. Thus, a system that can help collect text-based information and can export it into a digital format would benefit the medical community, reducing medical errors, burnout and improving efficiency of care. With the development of optical character recognition (OCR) techniques, the conversion of text within images to a digital format has become feasible. [3] Thus, leveraging of OCR has been identified as a potential pathway to produce a closed autonomous system that can digitize data from many patient care devices. Thus, in 2020, a group of innovators, among which were included the named inventors of the present application, embarked on the development of a system to facilitate at least some preliminary data capture from the display screen of a patient care device (intravenous pump).

[0008] The resulting prototype of that earlier work was documented in a 2021 journal publication. [4] Per that publication, the entirety of which is incorporated herein by reference, the prototype employed a consumer grade web camera, in combination with a laptop computer and OCR techniques, to digitize medical data (in a continuous fashion) that historically required manual recording by nurses (medication pump data). The prototype captured an image of the patient care device, used image manipulation techniques to optimize the image for OCR, then used OCR to automatically extract all of the text within the image. The extracted text was digitized, and the digitized information exported in real-time. Continuous data streams were generated through serially repeated image capture and real time processing of the serially captured images to generate the data stream from the OCR process in live time. In this early work, the protype implemented an entirely wired set-up, in which the camera and the computer were physically connected for wired data transfer. This early prototype was a proof of concept that demonstrated the workability of the overall idea, as a foundation for future work toward more practical, accurate and reliable implementations.

[0009] This past work was significantly limited in several critical areas at the time of publication back in 2021. First, the system was designed to only work on a single manufacturer brand and model of intravenous pump, which significantly limited its utility in future patient care environments. Further, the system could only work on an intravenous pump set-up, where intravenous pumps only represent a small portion of patient care devices that would benefit from continuous data capture (i.e. intravenous pumps, patient bedside vital sign monitors, feed pumps, other physiologic monitoring devices). Second, the system required one dedicated camera per device, and could not process multiple devices (either of same type, or different types) simultaneously with a single camera, given challenges with image segmentation, device identification, and data channel identification. Third, many factors associated with a clinical environment were not accounted for (including lighting, angle, resolution, monitor location or distance from camera), requiring further optimization of the algorithm to function in more challenging conditions. Fourth, the user-interface was designed only for one system in a fully wired implementation. If such a system were to be deployed in a clinical, forward operating, or austere patient care environment for data capture, there needed to be substantial improvements in the software / hardware construct to facilitate the option of entirely portable and wireless data transmission, to achieve an “edge computing” solution. Finally, various other factors that must be fulfilled for clinical operation had not been tested or evaluated (user error, optimal location, and hardware compatibility testing). Thus, it was apparent that aspects of the OCR and image manipulation implementation required more refinement to produce a highly functional, ubiquitous, and portable platform that could reach operational readiness and commercialization potential.

[0010] Since then, significant time and effort have been invested to advance the technology to address these shortcomings of the early prototype, and it is the novel and inventive improvements made thereto that are the subject of the present application. SUMMARY OF THE INVENTION

[0011] According to a first aspect of the invention, there is provided a computer vision system for continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said system comprising: a digital camera operable to capture digital imagery of said one or more display-equipped patient care devices; a computing device connected to said digital camera and comprising a transmitter for communicating data to an external device, and one or more processors and non-transitory computer readable memory coupled thereto, in which there are stored executable statements and instructions for execution by said one or more processors, which, when executed, cause performance of the following steps by said one or more processors:

[0012] (a) triggering capture, by said digital camera, of a digital image encompassing said one or more display-equipped patient care devices;

[0013] (b) triggering execution of one or more image processing algorithms that:

[0014] (i) perform edge detection on said captured image;

[0015] (ii) based on said edge detection, generate a set of bounding rectangles each framed around a respective imaged object located in the captured image;

[0016] (iii) for each of said bounding rectangles generated, evaluate image contents framed within the bounding rectangle therein against a plurality of unique class objects stored in a reference data repository, of which each unique class object represents a uniquely respective one of a plurality of classified display-equipped patient care devices denoting a candidate identity for the respective image object framed within the bounding rectangle; (iv) for any one of the imaged objects framed within the bounding rectangles that was successfully matched in step (b)(iii) to one of the classified display- equipped patient care devices, perform on said one of the imaged objects an image optimization routine that is particularly configured, in at least one aspect of image manipulation, based at last partly on which one of the classified display equipped patient care devices said one of the imaged objects was matched to, thereby deriving an optimized image of said one of the imaged objects that has been device-specifically optimized for optical character recognition (OCR); and

[0017] (v) subject said optimized image to an OCR process to extract text information therefrom; and

[0018] (c) transmitting to an external device output data representative of said text information garnered from the OCR process, which output data thereby denotes a real time reading of displayed text output of said any one of the imaged objects that was successfully matched in step (b)(iii).

[0019] According to a second aspect of the invention, there is provided a computer implemented method for continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said method comprising:

[0020] (a) capture of a digital image encompassing said one or more display- equipped patient care devices;

[0021] (b) execution of one or more image processing algorithms that:

[0022] (i) perform edge detection on said captured image;

[0023] (ii) generate a set of bounding rectangles each framed around a respective imaged object located in the captured image, at least in part by said edge detection; (iii) for each of said bounding rectangles generated, evaluate image contents framed within the bounding rectangle therein against a plurality of unique class objects stored in a reference data repository, of which each unique class object represents a uniquely respective one of a plurality of classified display-equipped patient care devices usable a candidate identity for the respective image object framed within the bounding rectangle;

[0024] (iv) for any one of the imaged objects framed within the bounding rectangles that was successfully matched in step (b)(iii) to one of the classified display- equipped patient care devices, perform on said one of the imaged objects an image optimization routine that is particularly configured, in at least one aspect of image manipulation, based at least partly on which one of the classified display equipped patient care devices said one of the imaged objects was matched to, thereby deriving an optimized image of said one of the imaged objects that has been device-specifically optimized for optical character recognition (OCR); and

[0025] (v) subject said optimized image to an OCR process to extract text information therefrom; and

[0026] (c) transmission of output data representative of said text information garnered from the OCR process, which output data thereby denotes a digitized transmission of displayed text outputted by said any one of the imaged objects that was successfully matched in step (b)(iii).

[0027] According to a third aspect of the invention, there is provided a system for use in continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said system comprising: an image processing apparatus comprising a communications network connection, one or more processors and non-transitory computer readable memory coupled thereto, in which there are stored executable statements and instructions for execution by said one or more processors, which, when executed, cause performance of the following steps by said one or more processors:

[0028] (a) receipt, through said communications network connection from an onsite visioning system operating in a same environment as said one or more display- equipped patient care devices, of a digital image encompassing said one or more display-equipped patient care devices;

[0029] (b) execution of the one or more image processing algorithms of steps of step (b) of any one of claims 1 to 13; and

[0030] (c) transmission, storage or display output data representative of said text information garnered from the OCR process, which output data thereby denotes a real time reading of displayed text output of said any one of the imaged objects that was successfully matched in step (b)(iii).

[0031] According to a fourth aspect of the invention, there is provided a computer implemented method for use in continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said method comprising the following steps executed one or more computer processors:

[0032] (a) receipt, over a data communications connection from an onsite visioning system operating in a same environment as said one or more display- equipped patient care devices, of a digital image encompassing said one or more display-equipped patient care devices;

[0033] (b) execution of the image processing algorithms of step (b) of any one of claims 14 to 26; and

[0034] (c) transmission, storage or display output data representative of said text information garnered from the OCR process, which output data thereby denotes a real time reading of displayed text output of said any one of the imaged objects that was successfully matched in step (b)(iii).

[0035] BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Preferred embodiments of the invention will now be described in conjunction with the accompanying drawings in which:

[0037] Figure 1 is a combined block and process diagram of one embodiment of an inventive computer vision system for continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices.

[0038] Figure 2 is a basic block diagram of hardware componentry of the computer vision system of Figure 1.

[0039] Figures 3A to 3D are screenshots of the display screens of different patient care devices, including a continuous drug infusion device administrating two drugs (ketamine and fentanyl) in Figure 3A, another such continuous drug infusion device administering a single drug (dopamine) in Figure 3D, an intracranial pressure measurement device in Figure 3B, and a key systemic physiology measurement device in Figure 30.

[0040] Figure 4 illustrates a suite of several bedside patient care devices having respective display screens that can be simultaneously monitored by the inventive computer vision system of Figure 1 to transform the visually displayed readout data of those patient devices into real-time transmissible data for remote monitoring and / or logging purposes.

[0041] Figures 5A through 5D respectively illustrate the display screens of four different patient care devices (two medical pumps in Figures 5A & 5B, a systemic physiology measurement device in Figure 5C, and an intracranial pressure measurement device in Figure 5D), each being displayed in combination with associated features by which the device is classified in an object classification repository of the software of the inventive computer vision system of Figure 1 .

[0042] Figures 6A through 6E visually illustrate image processing and manipulation steps performed by the inventive computer vision system of Figure 1 , starting from initial image capture in Figure 6A, performance of edge detection on the captured image in Figure 6B to form bounding rectangles around candidate areas of the image that may contain display screens of patient care devices, as shown in Figure 6C, performance of object classification against a data repository of pre-classified patient care devices using located features in the rectangularly bound candidate areas to identify patient care devices shown in the rectangularly bounded candidate areas, which identified patient care devices are shown as positively identified in Figure 6D, from which the imaged display monitors of those identified devices can be individuated, as shown in Figure 6E, for individualized further processing of those images in devicespecific fashion.

[0043] Figures 7A through 7D visually illustrate device-specific pre-OCR manipulation of four individuated images from Figure 6E according to four differently classified patient care devices from which those images were captured, demonstrating varying complexity of pre-OCR image manipulation for different devices, and further illustrating final automated refinement of extracted OCR text using a specialized medical OCR dictionary of the inventive computer vision system of Figure 1 .

[0044] Figures 8A and 8B illustrate timestamped real-time output data, in comma separated value (CSV) format, from the inventive computer vision system of Figure 1 , which output data is shown in accompaniment by the latest captured image of the system in a user interface thereof in Figure 8A, and instead shown in a generic spreadsheet program in Figure 8B, for example after communication of the output data from the inventive computer vision system to a remote data collection point.

[0045] DETAILED DESCRIPTION

[0046] In conception and reduction to practice of the present invention, a few key facets / problems were focussed upon, including A. Improved Image Manipulation Techniques (in the form of new device recognition, the “multiple device problem”, and optimal image adjustment methods for OCR that is more adaptable to real-world environments), B. development of a specialized OCR dictionary (focusing on specialized medical terms), and C. optimization of hardware systems to facilitate more portable and wireless “edge computing” capacity (aiding in future deployment of the technology in any environment). It should be noted that key code architecture was changed, with new focus on object-class oriented code and coding structure (leading to an entire ground-up rebuild from the published 2021 implementation). The focus of the inventors was to develop a system that is ready to be used for real-time data collection of text-based images, which can be deployed in a variety of clinical environments. The novel and inventive solutions to the above key problems are outlined in more detail below.

[0047] The integration of OCR systems for image capturing and text display requires refined and articulated images that are optimized for the collection of data being presented. In order to achieve an effectively accurate conversion system, there was a need to focus on three primary aspects of image manipulation (aka - SubChallenges): 1. individual device recognition, 2. multiple simultaneous device recognition, and 3. image adjustment optimized for OCR extraction.

[0048] Individual device recognition focuses on individualized external features and display characteristics inherent to each patient care device used within the clinical environment, including medication pumps, bedside patient vital sign monitors and other physiologic monitoring devices. In the context of OCR device detection, a unique problem exists for each individual make / model of patient care device. This is due to the fact that each patient care device has its own image quality, text, contrast, brightness and various other aspects associated with the respective display screen of that patient care device (demonstrated by Figure 3). Thus, each individual patient care device must be recognized, with critical data (drug identification, physiologic value, etc.) extracted from the larger image such that it can be sequentially assessed and evaluated using OCR technologies for future digitization (including name / lab, unit, and numerical value). This includes correctly identifying multiple channels of data in a single device display, for example multiple drug infusion channels in a single pump (Figure 3A), or multiple patient vital sign channels from a bedside display (Figure 3C).

[0049] Aside from being able to recognize different device makes / models in isolation, a fully functional system should ideally be able to utilize a single camera source to capture all devices in a single image simultaneously for digitization into a multi-channel data stream. This includes both being able to capture multiple devices of the same type with one image, and multiple devices of multiple different types simultaneously. A workable solution here was found to be the generation of custom classes to identify distinct devices in each image, correctly identify the device type of each distinct device, and identify all channels of visually displayed data within each device for digitation (i.e. name / label, units, and numerical value). Figure 4 provides a visual example of this challenge.

[0050] As emphasized by Figure 3, there are many aspects that are unique to the text of each patient care device, as well as the underlying relationships of that text to the broader environment of the patient care device (brightness, ambient room lighting and location (including angle / rotation of image and distance of camera from display, color, size)). These factors must be addressed when using OCR systems as they will significantly impact how the data is interpreted. As such, a workable system needs to address or account for these factors when modifying images, particularly if 100% accuracy is to be achieved through OCR manipulation and OCR techniques. Thus, this problem bears that each image needs to be vetted in its own unique way in order to effectively achieve the optimized collection of device displayed information.

[0051] Despite efforts and work with the evaluation and assessment of image manipulation techniques, there will always be outliers or factors that fall short in current uses of OCR for text conversion. Given the broad nature of the tasks that this system is intended to achieve, and to alleviate concerns and bolster the consistency and robustness of an ultimately implemented system, the creation of an OCR specialized dictionary has been undertaken in development of the inventive system. Factors like data names, types, and amounts can be directly assessed and related to the other collected information or known aspects to improve the consistency in system output. Such a specialized dictionary is vital to the overall optimization and assessment of the device.

[0052] Turning consideration to a general criteria by which acceptable solutions to the foregoing key problems should ideally be judged, the ultimately commercialized system must be fully ready for implementation within a clinical environment. As such the system must capture images using common devices and work on common hardware. A full working system will capture images and convert them to digital text. Breaking this down into smaller steps, this workflow is completed first by identifying all patient care devices in the image, after which each imaged patient care device identified is then analyzed and processed in preferably the following ways. First, the screen brightness, angle and contrast (or at least a subset thereof) are adjusted such that the image text will be optimized for extraction using OCR. After OCR extraction, the extracted text is adjusted, if necessary (i.e. character errors, or misspelling), according to a unique internal medical terminology reference dictionary / library, which includes a reference source for abbreviations (i.e. HR = heart rate, etc.) and unit symbols (i.e. mmHg, mL, etc). This ensures accurate and reproducible text extractions, reducing risk of transcriptional errors. Finally, these considerations are preferably wrapped up into a user-friendly software / hardware package, that the average person can implement for their own personal data collection.

[0053] Image manipulation is perhaps the most fundamental and important aspect of least some embodiments of the present invention, for which novel solutions are disclosed in three main areas: 1 ) the individual recognition of the different patient care devices, 2) simultaneous multiple device recognition, and 3) the optimization of the images for OCR processing. The importance of each of these factors is based on the critical aspects that are associated and vital to the optimal performance of this system, in that such systems need to be perfect or near perfect in this regard for the overall performance of the text extraction to achieve clinically usable accuracy. Leverage is made here primarily of object-oriented artifact intelligence, which as a summary, attempts to leverage abstract, encapsule, polymorph and inheritance to create code that is more robust to data changes, and simplistic in the implementation of these changes. Implementation of an objected-oriented approach necessitate a ground-up rebuild compared to the prior work published in 2021. The problems explained will only become exaggerated as more environments and patient care devices are added to the overall implementation of the system. Object-oriented artificial intelligence has the benefit of streamlining new code-based solutions and classification, offering avenues to implement new solutions seamlessly into the larger system. This present invention may leverage Python and Open Access Libraries including Os, Re, Sys, Difflib, Datetime, Time, Requests, Skimage. metric, PySide6, Open CV, Pillow, Pandas, Immutable, Easyocr, PaddleOCR, Tesseract and NumPy. These are nonlimiting examples, and ground-up custom-built OCR pipelines may alternatively be employed.

[0054] As demonstrated by Figure 5, every patient care device has at least one unique aspect to it that is vital to overall recognition and optimal performance. Such assessment is required given that each patient care device has its own unique text settings as well as brightness, lighting and other factors that greatly impact OCR and OCR manipulation. Thus, each patient care device may need to be treated differently and optimized in a respective way to ensure performance and verify overall accuracy. This is not to mention larger aspects that are associated with different devices where abbreviations of words and synopsis can be different between the devices. Finally, text and overall format should be adjusted, assessed and evaluated for each patient care device as they may impact overall performance. As an example of this is the difference between Figure 5C and 5D, which both indicate different physiological aspects, where Figure 5D is an intracranial pressure monitor and Figure 5C is a systemic blood pressure monitor. In the illustrated example, screen size is expressed as a ratio of screen width (W) to screen height (H). For each screen, a location data field is shown, which upon identification of a classified patient care device in a captured image is populated with a location of where that identified patient care device is located within the captured image, but in the illustration is populated with a TBD placeholder (meaning, to be determined).

[0055] A novel addition over the prior art is greater encompassing of various patient care device types, with their aforementioned features. In recoding of the prior software to adopt an object orientation implementation, each patient care device is now described as a unique class object that can be called as a reference and used to identify it from other devices. Thus, for every patient care device there is a repository record that can be accessed to ascertain key information about the patient care device and enable unique identification thereof. A resulting beneficial functionality is that more devices can be added seamlessly by addition to the repository of unique class objects, each denoting a respectively classified patient care device.

[0056] As visually denoted in Figure 6, and schematically denoted in Figure 1 , the following steps are executed in order to first isolate and optimize a respective image of each patient care device from the originally captured overall image. Initially the captured overall image is assessed for edges to develop a modified overall image that distinctly highlights the edges of that image (Figure 6B). The edges work by using the outline of different colors to highlight features. From the modified edges image, complete shapes (ones which have a complete black object surrounded by white) can be found and highlighted (e.g. using open CV) for rectangles bounding these shapes to be created (Figure 6C). In novel supplementation of these steps from the 2021 publication, is object-oriented classification of the patient care devices, performed from Figure 6C onward. As mentioned above, each patient care device has its unique features classified, thus each selected rectangle can be evaluated for these features to get from Figure 6C to 6D (searching each rectangle for the features described). Unlike past work that just focused on finding one patient care device, the inventive system searches from multiple devices, already classified in the repository. This novel storage of such repository of classified medical devices denotes significant change, and dramatic improvement, over the prior art. Thus, from the rectangles found in Figure 6C, each patient care device has its own unique features that allow it to be identified including location, size, display style and the text within the found rectangle. Through combined consideration of these features, each patient care device can be uniquely identified within the localized rectangles that can be used to locate various images (Figure 6D / E). From this, each rectangle is evaluated based on the selection process of the aforementioned factors for each patient care device and is used to determine the target of individual patient care devices themselves such that they can be individually optimized in the OCR process (described next). Thus, finalized extracted images for each patient care device can be seen in Figure 7E.

[0057] Once the individual patient care devices have been captured and are identified (Figure 6E), such individualized device images are then analyzed using OCR techniques to extract the key digital information (including multiple channels worth of data, if present in an individual device interface). Owing to the class-based methodology of identifying uniquely classified patient care devices, each individualized device image is adjusted according to the need of each patient care device. Within the stored class label for each device, the features stored therein include at least a subset thereof that are exploitable to optimally adjust the image. In one embodiment, each class has seven features (screen background color, screen size, location, critical text types for extraction (i.e. drug names, units, etc.), text volume (i.e. different devices display different volumes of critical text for extraction; some only numerical values and units, others drug names, etc.), text size, and text color), though if other features are required or desirable (i.e. screen shape), they can be easily added to the classifications of the classified patient care devices of the repository (given the class architecture). In one embodiment, the images from each patient care device are treated slightly differently, however they all have the same fundamental steps, such as those outlined in the prior art (examples of which are given in Figure 7)[4], These steps involve assessing captured device images for brightness / contrast (ensuring that the text is well separated from the background), angle / tilt / rotation of the image (ensuring text is correctly oriented from captured image) and text size / location / text quality (ensuring text is properly located and has the required quality and values to give a good result). While these particular aspects are employed in the present embodiment to contribute to accurate classification of a given patient care, this is a non-limiting and non-exhaustive list of examples.

[0058] Figure 7A provides an example of an image that needs no extra image manipulation, apart from conversion into a black and white screen, to get the required text and can be sampled directly. Figure 7B, shows a device that requires limited overall changes apart from conversion into a black and white screen. Figure 7C shows a device image that requires better localization and conversion of the image to black and white, which can be done with an adaptive threshold over the image. Figure 7D shows a device image that that needs some added rotation of the image which, which is done by finding localized edges and slightly rotating the image. With the finalized text extracted (including respective extracted text from all available channels from multichannel devices showing multiple channels of on-screen data), this can then be fed into the OCR dictionary for final text processing.

[0059] Once the system has effectively pulled the required text, there may be one or more aspects that need the modification or adjustment of the text to better resolve both characters and identified names / labels such that correct solutions in the digitization can be implemented. This is a common tactic in any sort of data manipulation where errors or missing values can occur. To enable this in the context of the present invention, development was made of a specialized library through the collection of medical terms, their associated patient care devices and key factors associated with them (amount, type, limits, etc). This includes a reference dictionary for medical term / names (i.e. to cross reference spelling and ensure no transcriptional character substitution errors - with “f” for “t” being a common example). Similarly, this library contains medical acronyms and abbreviations commonly seen on patient monitoring devices (i.e. HR = heart rate, ABP = arterial blood pressure, etc.), and mathematical / unit references to ensure drug dosing or physiology variable units are correctly transcribed (i.e. mmHg, mL, etc.). Such factors are used and contrasted against the collected image text that is extracted using the OCR methodologies, such that corrections or modifications to the final output are made automatically.

[0060] This OCR system consists of a living (dynamic) dictionary that can be manually modified depending on what is required for new devices added to the dynamic repository (given the object-oriented nature of the software code). This allows for the ongoing update of new drugs and medication a not originally present in an initially populated dictionary. Further, as demonstrated by Figure 7, there are some devices that do not identify a type or meaning of the on-screen data streaming information (Figure 7D), for example showing only an unlabelled unitless numerical readout, some devices whose on-screen information is too small to be effectively converted by OCR (Figure 7C), some situations where the word is incorrectly converted by OCR (Figure 7 A), or the converted information is misarranged (Figure 7B). A workable solution to all of these issues is the ability to modify how the information is streamed into the final data output, tagging extracted text and matching it with known units and rate affiliated with that particular device that are stored in our OCR library.

[0061] In the case for Figure 7C / D, the manual input or correction of missing or erroneously converted values can be completed on first such occurrence whose paired device classification doesn’t already contain such corrective values (given the nature of the new object-orientated code architecture), which manually updated device application will from thereon be implemented within the system for autonomously corrected data streaming in the future. For Figure 7A / B, first the OCR dictionary can be referenced which can auto-correct the spelling and assess which numbers correctly correspond to the data. To enable manual intervention, preferred embodiments have a user interface 50 that allows for a manual override of the errors and pairs this information with each monitor’s classification, which will improve the overall system performance.

[0062] Especially, though not exclusively, when combined together, the dynamically updatable dictionary and the dynamically updatable classification repository achieve a semi-automated process where automated aspects of data curation and data management can be bolstered by medical professionals and on-site researchers to provide an adaptive system that is collecting and dynamically updating information over time to improve system performance.

[0063] Preferred embodiments feature an enclosed wireless “edge computing” system, that captures and processes the images locally in the same environment as the medical device(s), and wirelessly transmits device data streams in a multichannel format to any remote computing infrastructure (i.e. laptop, central server or data repository), for example in a universal CSV (comma separated values) format. Such data can be easily integrated into any electronic patient or medical record. In a prototyped embodiment, a Raspberry PI was used to wirelessly stream still images captured by a connected camera onward to a laptop computer running a Python interface for device recognition, in demonstrating working of enclosed platform. This platform can be deployed in any clinical, forward operating or austere environment to facilitate wireless and autonomous data capture from any medical device display, as schematically shown in Figure 1 . Figure 1 schematically illustrate one preferred embodiment of the present invention, in which the inventive computer vision system 10 is employed in a local environment Ev (e.g. hospital room) occupied by a, typically though not necessarily bedded, patient 12, in association with which a collection of one or more display- equipped patient care devices 14A, 14B are operating in a patient-proximate (e.g. bedside) fashion in which they respective display screens are visually readable by medical professionals and / or other occupants / attendants of this patient environment. Aside from healthcare environments (hospitals, hospices, homecare, etc.) other examples of patient environments where the system 10 can be used include, without limitation, veterinary and military environments, the former of which also clarifies that the patient need not necessarily be human. The computer vision system 10 comprises a computing device 16 and a digital camera 18 of integrated or connected (wired or wirelessly) relationship to that computing device 16. The digital camera 18 is appropriately positioned and aimed to capture the patient care devices 14A, 14B within the camera’s field of vision.

[0064] Referring to Figure 2, the computing device 16, comprises one or more computer processors 20; a transmitter 22 (typically embodied in a two-way transceiver) for communicating output data from the computing device 16 to a remotely external device 24 (shown in Figure 1 ) situated remotely of the computing device’s operating environment through a direct or networked connection between those devices 16, 24; and volatile random access memory (RAM) 26 and non-volatile memory 28, both of which are non-transitory computer readable memory for reading and writing of data thereto and therefrom by the processor(s) 20. All such hardware componentry of the computing device 16, and the integrated or connected digital camera 18, are interconnected, directly or indirectly, by one or more busses 30 to enable any and all herein described cooperative functionality between these components. The nonvolatile memory 28 has stored therein inventive software of the computer vision system, which software is composed of executable statements and instructions for execution by the processor(s) 20 to perform any all algorithms, processes, routines, tasks, and steps described herein, except for any of those that may be explicitly described as being performed by another means or actor. In Figure 1 , the software is schematically denoted by functional blocks of a flow diagram schematically representative of the algorithmic workflow of the software, thought it will be appreciated that the software, while being implementable in modules, need not necessarily be implemented via modules of matching description to individuated functional blocks of the schematic illustration.

[0065] Referring again to Figure 1 , the digital camera 18 continually captures images that each capture the totality of patient care devices 14A, 14B in the camera’s field of view, which captured images are fed as input the algorithmic workflow of the software, as schematically shown at 32 and 44, respectively. Every captured image is processed in an analogous way, schematically illustrated by the first four functional blocks 46A - 46D of the Figure 1 workflow. First the captured image is reduced to identify only potential (candidate) patient care devices in the captured image through edge-detection, at first functional block 46A. At second functional block 46B, using a stored repository or catalogue 48 of classified patient care devices and the previously captured devices in a given image, the software selects, from among the potential patient care devices identified, actual (confirmed) patient care devices (including their location and device type) successfully matched to classified devices in the repository / catalogue 48, and individuates respective images of these actual (confirmed) patient care for further processing, and ignores all other potential patient care devices originally identified. At third functional block 46C, the individuated images of the selected actual devices from the captured image are optimized for OCR in a manner dependent on the classification of those selected actual devices. Finally, at functional block 46D, OCR processing of the individuated images is performed, extracting the text from all available data sources / channels in each individuated device image. These four major steps are described in more detail as follows.

[0066] In the first step at functional block 46A, edge-detection is performed to isolate all objects in the overall captured images that could potentially be display screens of patient care devices, of which there could be one or more in any single captured image. The edge-detection method identifies enclosed spaces of similar colours and localized shape, effectively reducing the captured image to one or more candidate devices or device-like objects. In data form, each of these identified enclosed objects includes information about a bounding rectangle that encloses the space that denotes a candidate device, and indicates a location at which this bounding rectangle and its enclosed space reside within the captured image. This information is used in the next step for further processing.

[0067] Step two at functional block 46B involves using the identified enclosed spaces and the repository / catalogue 48 of classified patient care devices to select appropriate patient care devices and types to assign to those enclosed spaces of the captured image according to the image contents of those enclosed spaces. The repository / catalogue 48 of classified devices contains identified features that make up each patient care device, and they are used to uniquely decern each potential / candidate device from identified bounding rectangles from the preceding step. Features that are used to catalogue each device in uniquely classified fashion may include all, or at least a subset of: display size, display colour, display text, display information, and display style. The software searches the enclosed objects, identifies all enclosed objects that each match a classified patient care device’s features in the classified catalogue 48. This allows the system to reduce the overall captured image to localized and individuated images of those successfully matched (confirmed) devices only, and label this localized and individuated image in accordance with a certain device type identified by the respective classification record in the repository / catalogue.

[0068] It should be noted that for new patient care devices introduced to the market after deployment of the inventive computer vision system 10, or existing patient care devices not initially catalogued in the repository 48 as thus equivalently “new” to the system, the software preferably includes an executable manual selection process that allows for the cataloguing and classifying of such new patient care devices in supplementation to the existing repository contents. This process involves a manual identification of a new patient care device, by either manually entering the features of the device (colour, text, size etc.) or by manually creating a bounding rectangle around the device within the captured image, from which the classification features will be automatically extracted. In the latter case of automated feature extraction, the software may seek to fill in the desired feature information by auto completing the full OCR process (as described below of functional block 46D) and then prompt the user for manual verification of the OCR results. Manual device feature identification methods (described above) for new devices would occur at the first instance of attempting to utilize the OCR pipeline. Once a new device is added to the system, the core text extraction steps will be conducted as previously described. For collection of user input in enablement of such user-update of a dynamically updatable repository 48, the illustrated embodiment of the computing device 16 in Figure 2 includes a user interface 50 that embodies both a graphical display and data input tools, for example optionally embodied in a touchscreen through which both display and user input are effected, or combination of a display screen and keypad, or combination of a display screen with one or more user input peripherals (keyboard, mouse, etc.). In the prototyped embodiment mentioned above, a laptop computer was used as the computing device, and thus inherently embodied both a display screen (optionally a touchscreen also usable for user input) and also user manipulatable input tools (keyboard, touchpad, optionally connected mouse).

[0069] Referring to the third functional block 46C of Figure 1 , after having reduced the captured image to localized and individuated images of classified devices only in the preceding functional block 46B, the software now prepares the localized and individuated images for optimized OCR of specifically optimized relation to the particular classified device of each individuated image. This is completed by using the stored features of the device in the catalogued repository 48 to adjust the localized and individuated image by changing one or more of its size, shape, colour, contrast, and various other parameters that the software deems optimal for text extraction from that particular classified device. This is completed based on internal device-specific feature catalogues built and stored within a reference library in the code base, which inform the algorithm which specific image manipulation processes are known to optimize the image for text extraction from a given identified device type. It should be noted that this step may, if required, be repeated one or more times if the final text extracted by the OCR process of a first or repeated iteration is determined to be incorrect (non-conformal to an expected output format prescribed by the classified patient device record in the repository 48), which conditional repetition is schematically illustrated by feed-back loop 52.

[0070] Within the second and third steps denoted by functional blocks 46B and 46C, classification of patient care device features is used, with this aspect leveraging a dynamically updatable catalogue of unique patient care devices that can be used in tandem with the other image manipulation techniques to improve the found text. Factors like device type inform the system on what information should be extracted and provide insights on how to evaluate the data (text information, amounts, and errors that can be corrected). Moreover, device-specific reference information stored within an internal developed library will inform optimal image manipulation techniques that should be applied to the image of a given medical device. This aspect of the prototyped embodiment of the system 10 provides a robust dynamic process to proactively account for challenges of real-world implementation and adaptation to new patient care devices not accounted for during repository creation for the demonstrated prototype.

[0071] Step four, at schematically shown functional block 46D, involves taking the localized and individuated image of the confirmed device that has now been preoptimized for the OCR process and performs the actual OCR processing. This OCR step is an image to text conversion process that extracts the key text shown on the display screen of the confirmed patient care device appearing in the individuated and pre-optimized image. The text is then cross referenced against what it should be anticipated to contain according to the details of the classified device as catalogued in the repository 48 (i.e. number of data channels, abbreviations commonly seen, medical terms commonly seen). If correct, then the image to text conversion is deemed complete, but if not, the software returns to the preceding step, as illustrated by feedback loop 52, for attempted re-optimization of the individuated image. The OCR process at block 46D is also bolstered by a customized medical text dictionary 54 that will make slight adjustments to the OCR extracted text against known medical terminology or device channel / signal abbreviation identification, thus achieving finalized output text at a final pre-transmission step of the process schematically denoted by functional block 46E of Figure 1. The medical text dictionary 54 is a living (dynamic) entity, that can receive regular updates to ensure the adaptability of the overall system 10 to new environments, patient care devices, and device combinations.

[0072] Step 46E denotes the end of local on-edge processing of the captured image data in the local patient environment by the computing device 16, from which the final text data output achieved at block 46E is then communicated, by the wireless transmitter 22 of the computing device 16, to the remotely external computing device 24 that serves to collect the outputted data from one of more of the inventive computer vision systems 10 for real-time display and / or data logging purposes of an ongoing stream of such output data. The remotely external computing device 24 may be embodied in a laptop, tablet, or desktop computer, a central server or cloud server, or other data repository, situated in a different and remote environment from the patient environment, whether that be another room or area of a same building or site as that patient environment, or at a different building or site situated distally remote of the patient-occupied building or site.

[0073] The communicated output data can then be used in for variety of clinical and administrative end-uses. First, live-time multi-device data collection, as described, eliminates the need for manual data input into medical records by healthcare personnel. This frees such individuals from such manually intensive tasks, reducing workloads, burnout and facilitating more time spent caring for patients. Further, this also reduces the risk of manual data transcription errors, reducing the chance of errors in medical care related to incorrectly transcribed data, improve safety in patient care. Second, such multi-channel temporally resolved data streams can be directly integrated with existing electronic medical records (EMR), ensuring continuous and accurate hands-free data transcription directly into medical repositories. Such data can then be used easily for workflow, administrative, and data safety audits, improving healthcare workflow efficiencies and care transparency. This includes tracking pharmacologic agent usage amounts and enabling hospital pharmacies the ability to ensure adequate drug stocks are maintained within the facilities. Third, patient care is currently transitioning to a more personalized approach in various settings. Such personalized care requires live-time integration of data from various data sources (i.e. pharmacologic and physiology), leveraging semi-automated or automated machine learning approaches to recommend timely bedside interventions and / or enable long-term patient trajectory predictions.

[0074] The multi-source data from the OCR system will facilitate transition to next-generation personalized care approaches. One such simple example would be integration of data from drug infusion devices with physiologic monitors, to allow for closed-loop drug administration to target specific physiology targets (i.e. such as a specific blood pressure). Finally, with the edge-computing and single camera for multidevice recognition capacities, this system could easily be employed in forward operating / austere environments (i.e. military, medical mission, NGO), completing the role of multiple medical personnel by automatically capturing all such device data and transmitting to both onsite and off-site medical staff. This would enable onsite staff to focus on critical medical care provision, while also facilitate off-site specialists / experts to be able to remotely assess the situation and provide remote virtual support / advice to the onsite personnel.

[0075] Since various modifications can be made in the invention as herein above described, and many apparently widely different embodiments of same made, it is intended that all matter contained in the accompanying specification shall be interpreted as illustrative only and not in a limiting sense. References:

[0076] 1. Aries MJ, Czosnyka M, Budohoski K, et al (2012) Continuous determination of optimal cerebral perfusion pressure in traumatic brain injury*. Critical Care Medicine 40:2456-2463. https: / / doi.Org / 10.1097 / CCM.0b013e3182514eb6

[0077] 2. Zeiler FA, Donnelly J, Menon DK, et al (2017) Continuous Autoregulatory Indices Derived from Multi-Modal Monitoring: Each One Is Not Like the Other. Journal of Neurotrauma 34:3070-3080. https: / / doi.org / 10.1089 / neu.2017.5129

[0078] 3. Carvalho MC (2016) Optical Character Recognition. In: Practical Laboratory Automation. John Wiley & Sons, Ltd, pp 207-209

[0079] 4. Froese L, Dian J, Batson C, et al (2021 ) Computer Vision for Continuous Bedside Pharmacological Data Extraction: A Novel Application of Artificial Intelligence for Clinical Data Recording and Biomedical Research. Frontiers in Big Data 4:74. https: / / d0i.0rg / l 0.3389 / fdata.2021 .689358

[0080] 5. Huiping Li, Doermann D, Kia O (2000) Automatic text detection and tracking in digital video. IEEE Transactions on Image Processing 9:147-156. https: / / doi.Org / 10.1109 / 83.817607

[0081] 6. Lim EHY, Liu JNK, Lee RST (2011 ) Text Information Retrieval. In: Lim EHY, Liu JNK, Lee RST (eds) Knowledge Seeker - Ontology Modelling for Information Search and Management: A Compendium. Springer, Berlin, Heidelberg, pp 27-36

[0082] 7. Ghai D, Jain N (2013) Text Extraction from Document Images- A Review. IJCA 84:40-48. https: / / doi.Org / 10.5120 / 14559-2661

[0083] 8. Li H, Doermann D (1999) Text enhancement in digital video using multiple frame integration. In: Proceedings of the seventh ACM international conference on Multimedia (Part 1 ) - MULTIMEDIA ’99. ACM Press, Orlando, Florida, United States, pp 19-22

Claims

CLAIMS:

1. A computer vision system for continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said system comprising: a digital camera operable to capture digital imagery of said one or more display-equipped patient care devices; a computing device connected to said digital camera and comprising a transmitter for communicating data to an external device, and one or more processors and non-transitory computer readable memory coupled thereto, in which there are stored executable statements and instructions for execution by said one or more processors, which, when executed, cause performance of the following steps by said one or more processors:(a) triggering capture, by said digital camera, of a digital image encompassing said one or more display-equipped patient care devices;(b) triggering execution of one or more image processing algorithms that:(i) perform edge detection on said captured image;(ii) based on said edge detection, generate a set of bounding rectangles each framed around a respective imaged object located in the captured image;(iii) for each of said bounding rectangles generated, evaluate image contents framed within the bounding rectangle therein against a plurality of unique class objects stored in a reference data repository, of which each unique class object represents a uniquely respective one of a plurality of classified display-equipped patient care devices denoting a candidate identity for the respective image object framed within the bounding rectangle;(iv) for any one of the imaged objects framed within the bounding rectangles that was successfully matched in step (b)(iii) to one of the classified display- equipped patient care devices, perform on said one of the imaged objects an image optimization routine that is particularly configured, in at least one aspect of image manipulation, based at last partly on which one of the classified display equipped patient care devices said one of the imaged objects was matched to, thereby deriving an optimized image of said one of the imaged objects that has been device-specifically optimized for optical character recognition (OCR); and(v) subject said optimized image to an OCR process to extract text information therefrom; and(c) transmitting to an external device output data representative of said text information garnered from the OCR process, which output data thereby denotes a real time reading of displayed text output of said any one of the imaged objects that was successfully matched in step (b)(iii).

2. The system of claim 1 wherein an attribute set of each unique class object comprises at least a subset of the following attributes:(a) a display size attribute representative of a display size of the respective one of the display-equipped patient care devices;(b) a background colour attribute representative of a display background colour of the respective one of the display-equipped patient care devices;(c) a text colour attribute representative of a displayed text colour of the respective one of the display-equipped patient care devices;(d) a drug text attribute representative of whether the respective one of the display-equipped patient care devices displays a drug name of a drug administered by said one of the display-equipped patient care devices;(e) a text size attribute representative of a size characteristic of text displayed by the respective one of the display-equipped patient care devices;(f) a text quantity attribute representative of a relative plenitude or sparsity of text displayed by the respective one of the display-equipped patient care devices; and(g) a channel quantity representative of a number of data channels displayed by the respective one of the display-equipped patient care devices.

3. The system of claim 2 wherein the attribute set of each unique class object comprises at least attribute (a).

4. The system of claim 2 or 3 wherein the attribute set of each unique class object comprises at least attribute (b).

5. The system of any one of claims 2 to 4 wherein the attribute set of each unique class object comprises at least attribute (c).

6. The system of any one of claims 2 to 5 wherein the attribute set of each unique class object comprises at least attribute (d).

7. The system of any one of claims 2 to 6 wherein the attribute set of each unique class object comprises at least attribute (e).

8. The system of any one of claims 2 to 7 wherein the attribute set of each unique class object comprises at least attribute (f).

9. The system of any one of claims 2 to 8 wherein the attribute set of each unique class object comprises at least attribute (g).

10. The system of any preceding claim wherein the executable statements and instructions are configured to repeat steps (b)(iv), (b)(v) and (c) on an ongoing basis, thereby generating and transmitting an ongoing stream of said output data reflective of ongoing text display output of said any one of the imaged objects thatwas successfully matched in step (b)(iii).11 . The system of any preceding claim wherein step (b) further comprises passing extracted text from said particularly configured OCR process through a specialized OCR dictionary populated with medical terminology, and the output data in step (c) is based at least partly on refinement of the extracted text by said specialized OCR dictionary.

12. The system of any preceding claim wherein step (b) further comprises evaluating the text information extracted against a benchmark reference defined at least partly by one or more attributes of the one of the classified display equipped patient care devices to which said one of the imaged objects was matched in attempted validation of the text information extracted.

13. The system of claim 12 wherein step (b), in instances of failed validation of the text information extracted, further comprises feedback guided repetition of step (b)(iv) in order to reoptimize the image and extract more accurate text information for revalidation.

14. A computer implemented method for continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said method comprising:(a) capture of a digital image encompassing said one or more display- equipped patient care devices;(b) execution of one or more image processing algorithms that:(i) perform edge detection on said captured image;(ii) generate a set of bounding rectangles each framed around a respective imaged object located in the captured image, at least in part by said edge detection;(iii) for each of said bounding rectangles generated, evaluate image contents framed within the bounding rectangle therein against a plurality of unique class objects stored in a reference data repository, of which each unique class object represents a uniquely respective one of a plurality of classified display-equipped patient care devices usable a candidate identity for the respective image object framed within the bounding rectangle;(iv) for any one of the imaged objects framed within the bounding rectangles that was successfully matched in step (b)(iii) to one of the classified display- equipped patient care devices, perform on said one of the imaged objects an image optimization routine that is particularly configured, in at least one aspect of image manipulation, based at least partly on which one of the classified display equipped patient care devices said one of the imaged objects was matched to, thereby deriving an optimized image of said one of the imaged objects that has been device-specifically optimized for optical character recognition (OCR); and(v) subject said optimized image to an OCR process to extract text information therefrom; and(c) transmission of output data representative of said text information garnered from the OCR process, which output data thereby denotes a digitized transmission of displayed text outputted by said any one of the imaged objects that was successfully matched in step (b)(iii).

15. The method of claim 14 wherein an attribute set of each unique class object comprises at least a subset of the following attributes:(a) a display size attribute representative of a display size of the respective one of the display-equipped patient care devices;(b) a background colour attribute representative of a display backgroundcolour of the respective one of the display-equipped patient care devices;(c) a text colour attribute representative of a displayed text colour of the respective one of the display-equipped patient care devices;(d) a drug text attribute representative of whether the respective one of the display-equipped patient care devices displays a drug name of a drug administered by said one of the display-equipped patient care devices;(e) a text size attribute representative of a size characteristic of text displayed by the respective one of the display-equipped patient care devices;(f) a text quantity attribute representative of a relative plenitude or sparsity of text displayed by the respective one of the display-equipped patient care devices; and(g) a channel quantity representative of a number of data channels displayed by the respective one of the display-equipped patient care devices.

16. The method of claim 15 wherein the attribute set of each unique class object comprises at least attribute (a).

17. The method of claim 15 or 16 wherein the attribute set of each unique class object comprises at least attribute (b).

18. The method of any one of claims 15 to 17 wherein the attribute set of each unique class object comprises at least attribute (c).

19. The method of any one of claims 15 to 18 wherein the attribute set of each unique class object comprises at least attribute (d).

20. The method of any one of claims 15 to 19 wherein the attribute set of each unique class object comprises at least attribute (e).21 . The method of any one of claims 15 to 20 wherein the attribute set of each unique class object comprises at least attribute (f).

22. The method of any one of claims 15 to 21 wherein the attribute set of each unique class object comprises at least attribute (g).

23. The method of any one of claims 14 to 22 comprising ongoing repetition steps (b)(iv), (b)(v) and (c) on an ongoing basis, thereby generating and transmitting an ongoing stream of said output data reflective of ongoing text display output of said any one of the imaged objects that was successfully matched in step (b)(iii).

24. The method of any one of claims 14 to 23 wherein step (b)(iv) comprises passing extracted text from said particularly configured OCR process through a specialized OCR dictionary populated with medical terminology, and the output data is based at least partly on refinement of the extracted text by said specialized OCR dictionary.

25. The method of any one of claims 14 to 24 wherein step (b) further comprises evaluating the text information extracted against a benchmark reference defined at least partly by one or more attributes of the one of the classified display equipped patient care devices to which said one of the imaged objects was matched in attempted validation of the text information extracted.

26. The method of claim 25 wherein step (b), in instances of failed validation of the text information extracted, further comprises feedback guided repetition of steps (b)(iv) and (b)(v) in order to reoptimize the image and extract more accurate text information for revalidation.

27. A system for use in continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said system comprising: an image processing apparatus comprising a communications networkconnection, one or more processors and non-transitory computer readable memory coupled thereto, in which there are stored executable statements and instructions for execution by said one or more processors, which, when executed, cause performance of the following steps by said one or more processors:(a) receipt, through said communications network connection from an onsite visioning system operating in a same environment as said one or more display- equipped patient care devices, of a digital image encompassing said one or more display-equipped patient care devices;(b) execution of the one or more image processing algorithms of steps of step (b) of any one of claims 1 to 13; and(c) transmission, storage or display output data representative of said text information garnered from the OCR process, which output data thereby denotes a real time reading of displayed text output of said any one of the imaged objects that was successfully matched in step (b)(iii).

28. A computer implemented method for use in continuous data extraction and digitization of displayed text from one or more display-equipped patient care devices, said method comprising the following steps executed one or more computer processors:(a) receipt, over a data communications connection from an onsite visioning system operating in a same environment as said one or more display- equipped patient care devices, of a digital image encompassing said one or more display-equipped patient care devices;(b) execution of the image processing algorithms of step (b) of any one of claims 14 to 26; and(c) transmission, storage or display output data representative of said textinformation garnered from the OCR process, which output data thereby denotes a real time reading of displayed text output of said any one of the imaged objects that was successfully matched in step (b)(iii).

29. Non-transitory computer readable memory having stored therein the executable statements and instructions recited in any one of claims 1 to 13 and 27.

Images