Transferring and displaying of breathing visualization data

Abstract

Implementations described herein disclose a method including determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient, communicating the respiratory parameter to a display apparatus, modulating a breathing visualization patch based on the respiratory parameter, and displaying the modulated breathing visualization patch.

Description

TRANSFERRING AND DISPLAYING OF BREATHING VISUALIZATION DATACROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 571,346, filed March 28, 2024, the entire content of which is incorporated herein by reference.BACKGROUND

[0002] Respiratory rate (RR) is one of the common vital signs that is measured in clinical settings. The levels of RR provide information about the health of patients. Similarly, any significant change in the levels of RR are often early indicators of major health complications such as respiratory tract infections, respiratory depression associated with opioid consumption, anesthesia and / or sedation, as well as respiratory failure. Patient’s RR can be measured and displayed in a number of different ways, including using a spirometer, pulse oximeters, etc. Health care facilities may disclose the patient’s RR using display devices in a number of difference settings and locations such as in central nursing room, on computer on wheels located in various different parts of hospital, on mobile devices, etc.SUMMARY

[0003] Implementations described herein disclose a method including determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient, communicating the respiratory parameter to a display apparatus, modulating a breathing visualization patch based on the respiratory parameter, and displaying the modulated breathing visualization patch.

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Other implementations are also described and recited herein.

[0006] Brief Descriptions of the Drawings

[0007] A further understanding of the nature and advantages of the present technology may be realized by reference to the figures, which are described in the remaining portion of the specification.

[0008] FIG. 1 illustrates an example implementation of a system for transferring and displaying of breathing visualization data as disclosed herein.

[0009] FIGS. 2A and 2B illustrate example image displayed using the system for transferring and displaying of breathing visualization data as disclosed herein.

[0010] FIGS. 3A and 3B illustrate alternative example images displayed using the system for transferring and displaying of breathing visualization data as disclosed herein.

[0011] FIG. 4 illustrates example operations of the system for transferring and displaying of breathing visualization data as disclosed herein.

[0012] FIG. 5 illustrates alternative example operations of the system for transferring and displaying of breathing visualization data as disclosed herein.

[0013] FIG. 6 shows a portable non-contact subject monitoring system that includes a non-contact detector and a computing device.

[0014] FIG. 7 shows a semi-portable non-contact subject monitoring system that includes a non-contact detector and a computing device.

[0015] FIG. 8 shows a non-portable non-contact subject monitoring system that includes a non-contact detector and a computing device.

[0016] FIG. 9 is a block diagram illustrating a system including a computing device, a server, and an image capture device.DETAILED DESCRIPTIONS

[0017] Respiratory rate (RR) is one of the common vital signs that is measured in clinical settings. RR may be indicated as the number of breaths by a patient over a time period, such as number of breaths per minute. The levels of RR provide information about the health of patients. Similarly, any significant change in the levels of RR are often early indicators of major health complications such as respiratory tract infections, respiratory depression associated with opioid consumption, anesthesia and / or sedation, as well as respiratory failure.

[0018] Therefore, it is important for healthcare providers to have proper visualization of the breathing by the patients. Specifically, even while a patient is in their room, it is useful to communicate the images and or videos of the patient’s breathing to various locations throughout hospital or healthcare facilities. For example, such patient images or videos may be needed in a monitoring room used by nurses, doctors, technicians, various mobile devices, computer on wheels, etc. However, communicating such patient breathing visualization images or videos across the healthcare setting consumes considerable bandwidth. Specifically, quite often healthcare facilities may not have the necessary infrastructure to communicate patient breathing visualization video feeds across its network.

[0019] The technology disclosed herein provide various implementations for communicating low bandwidth breathing visualization information across network that allows the receiving devices to reconstruct visualization of patient’s respiratory activity and / or status. For example, such low bandwidth information communicated across the network may be the respiratory rate. A receiving device, such as a monitor in a central nursing room, may use the respiratory rate (RR) to reconstruct a breathing visualization patch on a representation of a patient. In one implementation, the RR may be communicated intermittently, say every 10 seconds, to reduce the bandwidth usage even further.

[0020] Alternatively, a respiratory waveform may be part of the breathing visualization information communicated over the network. In such an implementation, the receiving device, such as a computer on wheels, may reconstruct a breathing visualization patch on a representation of a patient for display. For example, the receiving device may modulate a size of the breathing visualization based on amplitude of the respiratory waveform at given instance. Thus, when the respiratory waveform is at peak, the size of the breathing visualization is higher and on the other hand, when the respiratory waveform is at trough, the size of the breathing visualization is lower. Alternatively, intensity or color of the breathing visualization based on amplitude of the respiratory waveform. In another implementation, the phase of the respiratory waveform may be used to modulate the size, the color, the intensity, or some other display parameter of the breathing visualization.

[0021] Alternatively, the low bandwidth breathing visualization information communicated across the network may include the inhalation exhalation (EE) ratio along with the RR. Subsequently, the EE ratio may be used to modulate a display parameter of the breathing visualization. For example, if the breathing visualization is shown using multiplecolors, the breathing visualization maybe shown with green color for the inhalation period and with red color during the exhalation period. Alternatively, the intensity of the colors of the breathing visualization may be modulated using the amplitude of the respiratory waveform, if available.

[0022] In one implementation, a breathing visualization image is generated based on the breathing signal, such as breathing volume, of the patient and a compressed version of the breathing visualization image is communicated over the network to the display devices. Alternatively, the breathing visualization image maybe binarized to include information including height of the breathing visualization image, the horizontal beginning and ending of each horizontal line of the breathing visualization image, etc. Furthermore, the binary information may be encoded using run-length encoding to remove redundant information from the binarized breathing visualization image data. Subsequently, such binarized breathing visualization image data is communicated over the communication network and the receiving device may reconstruct a breathing visualization patch for display. Furthermore, the receiving device may also use the RR or the respiratory waveform to modulate the color, intensity, or other display parameter of the reconstructed breathing visualization.

[0023] In one implementation, the breathing visualization image is downsized to remove high-frequency features that are less important or that are not visually discernable to the user. Furthermore, the compressed or downsized version of the breathing visualization image may also be sent only intermittently, such as every Is, 10s, 60s, etc. In one implementation, the breathing visualization system evaluates the breathing visualization image intermittently to determine if there are any significant changes to the breathing visualization image and communicates the binarized version of the breathing visualization image only when there is a significant change. For example, if the size of the breathing visualization image increases or decreases over, say 10%, the binarized version of the breathing visualization image is re-communicated. Alternatively, the binarized version of the breathing visualization image is re-communicated if the rate of change of the volume of the patient’s breathing signal is above a threshold.

[0024] In an alternative implementation, together with the breathing visualization image, the RR, and / or the respiratory waveform, an image of the patient is also communicated intermittently. The communication of the image of the patient may be evenless frequently / more intermitently than the communication of the breathing visualization image, the RR, and / or the respiratory waveform. For example, the image of the patient may only be communicated in response to determining that the patient has moved in the bed. Such patient movement information may be generated from analyzing the images received from a camera. Communicating the image of the patient in bed is useful as it may impact the breathing patern of the patient.

[0025] FIG. 1 illustrates a non-contact subject monitoring system 100 for a patient 102. The system 100 includes a non-contact detector system 110 placed remote from the patient 102. In this embodiment, the detector system 110 includes a camera system 114, particularly, a camera that may include an infrared (IR) detection feature. The camera system 114 may be a depth sensing camera system, such as a Kinect camera from Microsoft Corp. (Redmond, Washington) or a RealSenseTM D415, D435 or D455 camera from Intel Corp. (Santa Clara, California).

[0026] The camera system 114 is remote from the patient 102, in that it is spaced apart from and does not physically contact the patient 102. The camera system 114 may be positioned in close proximity to or on the bed of the patient 102. The camera system 114 has a field of view F that encompasses at least a portion of the patient 102. The field of view F may be selected to be at least the torso of the patient 102. The camera system 114 includes a depth sensing camera that can detect a distance between the camera system 114 and obj ects in its field of view F. Such information can be used to determine that the patient 102 is within the field of view of the camera system 114 and determine a region of interest (ROI) to monitor on the subject. The ROI may be the entire field of view F or may be less than the entire field of view F. Once an ROI is identified, the distance to the desired feature is determined and the desired measurement(s) can be made.

[0027] The measurements (e.g., one or more of depth signal, RGB reflection, light intensity) are sent to a computing device 120 through a wired or wireless connection 121. The computing device 120 includes a processor 124 and memory 126 for storing data, software, computer instructions, etc. Sequential image frames of the patient 102 are recorded by the video camera system 114 and sent to the computing device 120 for analysis by the processor 124. Other embodiments of the computing device 120 may have different, fewer, or additional components than shown in FIG. 1. In some embodiments, the computing device 120 may be a server. In other embodiments, the computing device 120 ofFIG. 1 may be connected to a server. The captured images (e.g., still images or video) can be processed or analyzed at the computing device 120 and / or at the server to create a topographical map or image to identify the patient 102 and any other objects within the ROI.

[0028] The signals collected from the camera system 114 may be stored in the memory 126. For example, the signals from the camera system 114 may be stored as a depth image data stream 132. For example, the depth image data stream 132 may include depth of frames as captured by the camera system. Alternatively, the depth image data stream 132 may include depth of frames in the form or RGB signals, infra-red (IR) signals, etc.

[0029] Furthermore, the memory 126 may store various computer programs, software, instructions, etc., to process the data including the depth image data stream 132. In one implementation, the depth image data stream 132 may be processed to generate a breathing signal 134. The breathing signal 134 may be in the form of a flow signal representing the volume of breathing by the patient 102 measured in terms of ml of air breathed over a period such as ml / sec. Alternatively, the breathing signal 134 may be in the form of a volume signal generated as an integral of the flow signal over a segment of time.

[0030] The memory 126 may also store a breathing visualization system 136 that is configured to process the depth image data stream 132 and / or the breathing signal 134 to generate various breathing visualization information 140 that can be communicated over a network. Specifically, the breathing visualization system 136 may include various programs and algorithms that can be processed on the processor 124 to generate the breathing visualization information 140. Specifically, the breathing visualization information 140 may be low-bandwidth information such that it can be communicated over a network while utilizing lower amount of network bandwidth.

[0031] For example, the breathing visualization information 140 may include various respiratory rate (RR) parameters 142 including patient’s RR, patient’s inhalation exhalation (I: E) ratio, etc. Here the RR indicates the number of breaths taken by the patient per minute. The I:E ratio may indicate the proportions of each breath cycle devoted to the inspiratory and expiratory phases. The breathing visualization information 140 may also include a respiratory waveform 144 of the patient. For example, the respiratory waveform 144 may be a graphical description of the breathing by the patient that depicts the volume of breath inhaled and exhaled by the patient over time. The RR parameters 142 and / or the respiratorywaveform 144 are communicated to a display device 150 that uses this information to reconstruct a breathing visualization patch 168.

[0032] The display device 150 may be a computing device, such as a computer on wheels, a computing device used in a central nursing room, a mobile device used by a physician, a nurse, etc. The display device 150 may include memory 152, a processor 154, and a display 160. The memory 152 may include various computer programs and / or instructions to receive the breathing visualization information 140 and reconstruct the breathing visualization patch 168 that is displayed on an image 162 of a patient. As illustrated herein, the display 160 may display the image 162 of the patient, a respiratory waveform 164, and a RR 166. Specifically, the breathing visualization patch 168, constructed using the breathing visualization information 140 may be displayed on a torso of the image 162 of the patient.

[0033] Furthermore, the device 150 may modulate a display parameter of the breathing visualization patch 168 using the breathing visualization information 140. In one implementation, the color of the breathing visualization patch 168 on the display 160 may be modulated based on the respiratory waveform 144. For example, when the respiratory waveform 144 has a positive slope, indicating inhalation by the patient, the color of the breathing visualization patch 168 may be green and when the respiratory waveform 144 has a negative slope, indicating exhalation by the patient, the color of the breathing visualization patch 168 may be red. Alternatively, the brightness or intensity of the breathing visualization patch 168 may be modulated based on the respiratory waveform 144. In this case, as the respiratory waveform 144 indicates increasing amplitude, the brightness of the breathing visualization patch 168 is increased until the peak value and as the respiratory waveform 144 indicates decreasing amplitude, the brightness of the breathing visualization patch 168 is decreased until the trough value. Furthermore, in one implementation, both the color and the brightness or intensity of the breathing visualization patch 168 are modulated based on the respiratory waveform 144. Reconstructing the breathing visualization patch 168 modulated based on the respiratory waveform 144 provides a realistic visualization of the breathing by the patient on the display device 150 even when only the respiratory waveform 144, which requires substantially low communication bandwidth, are communicated over the network.

[0034] Alternatively, one or more display parameters of the breathing visualization patch 168 may be modulated based on the RR parameters such as the RR and the I:E ratio. For example, if the RR indicates that the patient takes ten breaths per minute giving each breath an average of six seconds with the EE ratio of 2: 1, on average each breath has four seconds of inhalation and two seconds of exhalation. In this case, the display device 150 may modulate the breathing visualization patch 168 for the four seconds of inhalation period with a green color and the two seconds of exhalation period with red color. Reconstructing the breathing visualization patch 168 modulated using the RR and the EE ratio allows the display device 150 to provide a realistic visualization of the breathing by the patient even when only the RR and the EE ratio, which requires substantially low communication bandwidth, are communicated over the network.

[0035] In one implementation, the breathing visualization information 140 also includes a breathing visualization image 146. Specifically, the breathing visualization image 146 may be a compressed version of a breathing visualization image generated based on the breathing signal 134. Alternatively, the breathing visualization image 146 may include binarized information including height of the breathing visualization image 146, the horizontal beginning and ending of each horizontal line of the breathing visualization image 146, etc. Furthermore, the binary information may be encoded using run-length encoding to remove redundant information from the binarized breathing visualization image data. Subsequently, such binarized breathing visualization image data is communicated over the communication network and the receiving device may use the binarized breathing visualization image data to reconstruct the breathing visualization patch 168 for display.

[0036] In one implementation, the breathing visualization image 146 is downsized to remove high-frequency features that are less important or that are not visually discernable to the user. Furthermore, the compressed or downsized version of the breathing visualization image 146 may also be sent only intermittently, such as every Is, 10s, 60s, etc. In one implementation, the breathing visualization system 136 evaluates the breathing visualization image 146 intermittently to determine if there are any significant changes to the breathing visualization image 146 and communicates the binarized version of the breathing visualization image 146 only when there is a significant change. For example, if the size of the breathing visualization image 146 increases or decreases over, say 10%, the binarized version of the breathing visualization image 146 is re-communicated.Alternatively, the binarized version of the breathing visualization image 146 is recommunicated if the rate of change of the volume of the patient’s breathing signal is above a threshold. The display device 150 may, upon receiving the binarized version of the breathing visualization image 146, reconstruct the breathing visualization patch 168 and display it on torso of the image 162 of the patient.

[0037] FIG. 2A indicates patient images 200 displayed using the system for transferring and displaying of breathing visualization data as disclosed herein. For example, the images 200 may be displayed on display in a central nursing station, a mobile device of a physician, a computer on wheel located in an emergency monitoring room of a healthcare facility, etc. Specifically, FIG. 2A illustrates an image of a patient 202A with a breathing visualization patch 204A super-imposed on the torso portion of the image of a patient 202A. The intensity of the breathing visualization patch 204A may be determined, for example, based on the amplitude of breathing volume as indicated by a respiratory waveform 206A. As illustrated, at time 208A, the amplitude of breathing volume as indicated by the respiratory waveform 206A is low and therefore, the intensity of the breathing visualization patch 204A is lower.

[0038] Compared to FIG. 2A, the breathing visualization patch 204B illustrated in FIG. 2B is at time 208B. As illustrated, the amplitude of breathing volume as indicated by the respiratory waveform 206B at time 208B is higher and therefore, the intensity of the breathing visualization patch 204B for the patient image 10 IB is higher. Herein, the varying intensities of the breathing visualization patches 204A and 204B are illustrated by the darker shade of the breathing visualization patch 204B compared to the lighter shade of the breathing visualization patch 204A.

[0039] While in the above illustrations, the intensities of the breathing visualization patches 204A and 204B are shown to vary based on amplitude of breathing volume as indicated by the respiratory waveform at given time, in alternative implementation, the color of the breathing visualization patches 204A and 204B may depend on the amplitude of breathing volume as indicated by the respiratory waveform at given time. Yet alternatively, the brightness and / or intensities of the breathing visualization patches 204A and 204B may be varied based on the phase of the respiratory waveform at given time. Thus, when the color of the breathing visualization patches 204A may be green when the phase of the respiratory waveform 206A indicates a positive slope at time 208A. On the other hand, the color of the breathing visualization patches 204B may be red when the phase of therespiratory waveform 206B indicates a negative slope at time 208B. Yet alternatively, the size of the breathing visualization patches 204A and 204B May also vary based on amplitude or phase of breathing volume as indicated by the respiratory waveforms 206A, 206B at given time.

[0040] The images 200 illustrated in FIGS. 2A and 2B provide realistic presentations of the breathing by a patient even when only the respiratory waveform 206 (206A and 206B) is communicated over the network to a display device displaying the images 200. This allows low bandwidth communication over the network, while still providing quite realistic presentation of breathing by the patient.

[0041] FIGS. 3A and 3B illustrate alternative images 300 displayed using the system for transferring and displaying of breathing visualization data as disclosed herein. Specifically, FIGS. 3 A and 3B illustrate animated version of the breathing visualization patches 302A and 302B on cartoon patient images 304A and 304B. Here the size of the breathing visualization patches 302A and 302B are varied based on the amplitude of breathing volume as indicated by respiratory waveforms 306A and 306B. Specifically, the size of the breathing visualization patches 302A is smaller when the amplitude of breathing volume as indicated by the respiratory waveform 306A at time 308A is low. Compared to that the size of the breathing visualization patches 302B is larger when the amplitude of breathing volume as indicated by the respiratory waveform 306B at time 308B is high.

[0042] FIG. 4 illustrates operations 400 of the system for transferring and displaying of breathing visualization data as disclosed herein. An operation 402 computes a respiratory waveform and / or respiratory parameters of a patient based on depth images of the patient. For example, the respiratory waveform may disclose the amplitude of breathing volume as a function of time. The respiratory parameters may include a respiratory rate (RR) and an inhalation exhalation (I:E) ratio. Subsequently, an operation 404 transmits the respiratory waveform and / or the respiratory parameters to a display device over a communication network. In one implementation, the operation 404 may transmit only the respiratory waveform. Alternatively, the operation 404 may transmit only one or more of the respiratory parameters.

[0043] An operation 406 constructs a modulated breathing visualization patch using the respiratory waveform and / or the respiratory parameters. For example, the operation 406 may modulate the intensity of the visualization patch based on the amplitude of breathingvolume as indicated by the respiratory waveform. Alternatively, the operation 406 may modulate the color of the visualization patch based on the amplitude of breathing volume as indicated by the respiratory waveform. In one implementation, the operation 406 may modulate the intensity of the visualization patch based on value of the I:E ratio. Yet alternatively, the operation 406 may modulate the intensity of the visualization patch based on value of the RR. Subsequently, an operation 408 superimposes the modulated breathing visualization patch on a representation of a patient and display the superimposed representation on a display device.

[0044] FIG. 5 illustrates alternative operations 500 of the system for transferring and displaying of breathing visualization data as disclosed herein. Specifically, an operation 502 generate a representation of a breathing mask based on images received from a depth camera. For example, the breathing mask can represent change in the volume of air in the lungs of a patient based on breath inhalation and exhalation as determined from the image data. For example, such breathing mask may be a two-dimensional representation of the amount of air the lungs, the size of lungs as inflated or deflated based on the inhalation: exhalation, etc.

[0045] An operation 504 binarizes the representation of the breathing mask. For example, such binarization may include generating height of the breathing visualization mask, the horizontal beginning and ending of each horizontal line of the breathing visualization image mask, etc. An operation 506 determines if there are any significant changes in the breathing by the patient as indicated by changes in the representation of the breathing mask. For example, if the size of the representation of the breathing mask reduces by more than 20%, indicated a substantial drop in the volume of air breathed by the patient, the operation 506 may determine it to be a significant change.

[0046] If so, an operation 508 encodes the binarized representation of the breathing mask. For example, such encoding may involve run-length encoding to remove redundant information from the binarized breathing visualization image data. Subsequently, an operation 510 communicates the encoded binarized representation of the breathing mask to display devices over a communication network. One or more display devices, such as a mobile device of a physician, may use the encoded binarized representation of the breathing mask to reconstruct a breathing mask patch and superimpose the reconstructed breathing mask patch on a representation image of the patient. An operation 512 displays therepresentation image of the patient with the reconstructed breathing mask patch superimposed thereon on a display.

[0047] FIG. 6 shows a portable non-contact subject monitoring system 600 that includes a non-contact detector 610 and a computing device 620. In this embodiment, the non-contact detector 610 and the computing device 620 are generally fixed in relation to each other and the system 600 is readily moveable in relation to the subject to be monitored. The detector 610 and the computing device 620 are supported on a trolley or stand 602, with the detector 610 on an arm 604 that is pivotable in relation to the stand 602 as well as adjustable in height. The system 600 can be readily moved and positioned where desired.

[0048] The detector 610 includes a first camera 614 and a second camera 615, at least one of which includes an infrared (IR) camera feature. The detector 610 also includes an IR projector 616, which projects individual features (e.g., dots, crosses or Xs, lines, or a featureless pattern, or a combination thereof etc.).

[0049] The detector 610 may be wired or wireless connected to the computing device 620. The computing device 620 includes a housing 621 with a touch screen display 622, a processor (not seen), and hardware memory (not seen) for storing software and computer instructions.

[0050] FIG. 7 shows a semi-portable non-contact subject monitoring system 700 that includes a non-contact detector 710 and a computing device 720. In this embodiment, the non-contact detector 710 is in a fixed relation to the subject to be monitored and the computing device 720 is readily moveable in relation to a subject lying on a bed 730. Specifically, the bed 730 may have a headboard 732, a side rail 734, and a mattress 736.

[0051] The detector 710 is supported on an arm 701 that is attached to a bed, in this embodiment, a hospital bed, although the detector 710 and the arm 701 can be attached to a crib, a bassinette, an incubator, an isolette, or other bed-type structure. In some embodiments, the arm 701 is pivotable in relation to the bed as well as adjustable in height to provide for proper positioning of the detector 710 in relation to the subject.

[0052] The detector 710 may be wired or wireless connected to the computing device 720, which is supported on a moveable trolley or stand 702. The computing device 720 includes a housing 721 with a touch screen display 722, a processor (not seen), and hardware memory (not seen) for storing software and computer instructions.

[0053] FIG. 8 shows a non-portable non-contact subject monitoring system 800 that includes a non-contact detector 810 and a computing device (not seen in FIG. 8). In this embodiment, at least the non-contact detector 810 is generally fixed in a location, configured to have the subject to be monitored moved into the appropriate position to be monitored.

[0054] The detector 810 is supported on a stand 801 that is free standing, the stand having a base 803, a frame 805, and a gantry 807. The gantry 807 may have an adjustable height, e.g., movable vertically along the frame 805, and may be pivotable, extendible and / or retractable in relation to the frame 805. The stand 801 is shaped and sized to allow a bed or bed-type structure to be moved (e.g., rolled) under the detector 810.

[0055] FIG. 9 is a block diagram illustrating a system including a computing device 900, a server 925, and an image capture device 985 (e.g., a camera, e.g., the camera system 114). In various embodiments, fewer, additional and / or different components may be used in the system.

[0056] The computing device 900 includes a processor 915 that is coupled to a memory 905. The processor 915 can store and recall data and applications in the memory 905, including applications that process information and send commands / signals according to any of the methods disclosed herein. In one implementation, the memory 905 may be non- transitory computer readable memory configured to store various computer-executable instructions that are executable on the processor or processor unit 935. Alternatively, the memory 905 may be physical article of manufacture that includes one or more computer- readable storage media encoding computer-executable instructions for executing on a computer system a computer process. The processor 915 may also display objects, applications, data, etc. on an interface / display 910 and / or provide an audible alert via a speaker 912. The processor 915 may also or alternately receive inputs through the interface / display 910. The processor 915 is also coupled to a transceiver 920. With this configuration, the processor 915, and subsequently the computing device 900, can communicate with other devices, such as the server 925 through a connection 970 and the image capture device 985 through a connection 980. For example, the computing device 900 may send to the server 925 information determined about a subject from images captured by the image capture device 985, such as depth information of a subject or object in an image.

[0057] The server 925 also includes a processor 935 that is coupled to a memory 930 and to a transceiver 940. The processor 935 can store and recall data and applications in the memory 930. With this configuration, the processor 935, and subsequently the server 925, can communicate with other devices, such as the computing device 900 through the connection 970.

[0058] The computing device 900 may be, e.g., the computing device 120 of FIG. 1. Accordingly, the computing device 900 may be located remotely from the image capture device 985, or it may be local and close to the image capture device 985 (e.g., in the same room). The processor 915 of the computing device 900 may perform any or all of the various steps disclosed herein. In other embodiments, the steps may be performed on a processor 935 of the server 925. In some embodiments, the various steps and methods disclosed herein may be performed by both of the processors 915 and 935. In some embodiments, certain steps may be performed by the processor 915 while others are performed by the processor 935. In some embodiments, information determined by the processor 915 may be sent to the server 925 for storage and / or further processing.

[0059] The devices shown in the illustrative embodiment may be utilized in various ways. For example, either or both of the connections 970, 980 may be varied. For example, either or both the connections 970, 980 may be a hard-wired connection. A hard-wired connection may involve connecting the devices through a USB (universal serial bus) port, serial port, parallel port, or other type of wired connection to facilitate the transfer of data and information between a processor of a device and a second processor of a second device. In another example, one or both of the connections 970, 980 may be a dock where one device may plug into another device. As another example, one or both of the connections 970, 980 may be a wireless connection. These connections may be any sort of wireless connection, including, but not limited to, Bluetooth connectivity, Wi-Fi connectivity, infrared, visible light, radio frequency (RF) signals, or other wireless protocols / methods. For example, other possible modes of wireless communication may include near-field communications, such as passive radio-frequency identification (RFID) and active RFID technologies. RFID and similar near-field communications may allow the various devices to communicate in short range when they are placed proximate to one another. In yet another example, the various devices may connect through an internet (or other network) connection. That is, one or both of the connections 970, 980 may represent several different computing devices and networkcomponents that allow the various devices to communicate through the internet, either through a hard-wired or wireless connection. One or both of the connections 970, 980 may also be a combination of several modes of connection.

[0060] The configuration of the devices in FIG. 9 is merely one physical system on which the disclosed embodiments may be executed. Other configurations of the devices shown may exist to practice the disclosed embodiments. Further, configurations of additional or fewer devices than the ones shown in FIG. 9 may exist to practice the disclosed embodiments. Additionally, the devices shown in FIG. 9 may be combined to allow for fewer devices than shown or separated such that more than the three devices exist in a system. It will be appreciated that many various combinations of computing devices may execute the methods and systems disclosed herein. Examples of such computing devices may include other types of infrared cameras / detectors, night vision cameras / detectors, other types of cameras, radio frequency transmitters / receivers, smart phones, personal computers, servers, laptop computers, tablets, RFID enabled devices, or any combinations of such devices.

[0061] In contrast to tangible computer-readable storage media, intangible computer- readable communication signals may embody computer readable instructions, data structures, program modules or other data resident in a modulated data signal, such as a carrier wave or other signal transport mechanism. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0062] The implementations described herein are implemented as logical steps in one or more computer systems. The logical operations may be implemented (1) as a sequence of processor-implemented steps executing in one or more computer systems and (2) as interconnected machine or circuit modules within one or more computer systems. The implementation is a matter of choice, dependent on the performance requirements of the computer system being utilized. Accordingly, the logical operations making up the implementations described herein are referred to variously as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations may be performed inany order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0063] A method disclosed herein provides determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient, communicating the respiratory parameter to a display apparatus, modulating a breathing visualization patch based on the respiratory parameter, and displaying the modulated breathing visualization patch.

[0064] A system disclosed herein includes memory, one or more processor units, and a breathing visualization system stored in the memory and executable by the one or more processor units, the breathing visualization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process including determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient, communicating the respiratory parameter to a display apparatus, modulating a breathing visualization patch based on the respiratory parameter, and displaying the modulated breathing visualization patch.

[0065] A physical article of manufacture disclosed herein includes one or more tangible computer-readable storage media encoding computer-executable instructions for executing on a computer system a computer process to determine respiratory rate of a patient, the computer process including determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient, communicating the respiratory parameter to a display apparatus, modulating a breathing visualization patch based on the respiratory parameter, and displaying the modulated breathing visualization patch.

[0066] The above specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the invention. Since many implementations of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended. Furthermore, structural features of the different embodiments may be combined in yet another implementation without departing from the recited claims.

[0067] The following examples are illustrative of the techniques described herein.

[0068] Example 1. A method, comprising: determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient; communicating the respiratory parameter to a display apparatus; modulating a breathing visualization patch based on the respiratory parameter; and displaying the modulated breathing visualization patch.

[0069] Example 2. The method of Example 1, wherein the respiratory parameter is a respiratory rate (RR) of the patient.

[0070] Example 3. The method of Example 1, wherein breathing visualization patch is displayed on a representation of torso of the patient.

[0071] Example 4. The method of Example 1, further comprising: determining, based on the image signal received from the camera focused on at least a portion of a patient a respiratory waveform of the patient; and communicating the respiratory waveform of the patient to the display apparatus.

[0072] Example 5. The method of Example 4, wherein modulating the breathing visualization patch further comprising, modulating brightness of the breathing visualization patch based on an amplitude of a respiratory waveform of the patient.

[0073] Example 6. The method of Example 4, wherein modulating the breathing visualization patch further comprising, modulating color of the breathing visualization patch based on an amplitude of a respiratory waveform of the patient.

[0074] Example 7. The method of Example 4, wherein modulating the breathing visualization patch further comprising, modulating at least one of color of the breathing visualization patch and brightness of the breathing visualization patch based on a phase of a respiratory waveform of the patient.

[0075] Example 8. The method of Example 2, wherein the respiratory parameter further including an inhalation exhalation (EE) ratio of the patient, and wherein modulating the breathing visualization patch based on the respiratory parameter further comprising modulating intensity of the breathing visualization patch based on the respiratory parameter.

[0076] Example 9. The method of Example 1, further comprising: generating a representation of a breathing mask for the patient; binarizing the representation of the breathing mask; encoding the representation of the breathing mask; and modulating intensity of the first color and intensity of the second color based on the respiratory parameter.

[0077] Example 10. A system comprising: memory; one or more processor units; a breathing visualization system stored in the memory and executable by the one or more processor units, the breathing visualization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising: determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient; communicating the respiratory parameter to a display apparatus; modulating a breathing visualization patch based on the respiratory parameter; and displaying the modulated breathing visualization patch.

[0078] Example 11. The system of Example 10, wherein the respiratory parameter is a respiratory rate (RR) of the patient.

[0079] Example 12. The system of Example 10, wherein the computer process further comprising: determining, based on the image signal received from the camera focused on at least a portion of a patient a respiratory waveform of the patient; and communicating the respiratory waveform of the patient to the display apparatus.

[0080] Example 13. The system of Example 12, wherein modulating the breathing visualization patch further comprising, modulating brightness of the breathing visualization patch based on an amplitude of a respiratory waveform of the patient.

[0081] Example 14. The system of Example 12, wherein modulating the breathing visualization patch further comprising, modulating color of the breathing visualization patch based on an amplitude of a respiratory waveform of the patient.

[0082] Example 15. The system of Example 12, wherein modulating the breathing visualization patch further comprising, modulating at least one of color of the breathing visualization patch and brightness of the breathing visualization patch based on a phase of a respiratory waveform of the patient.

[0083] Example 16. The system of Example 11, wherein the respiratory parameter further including an inhalation exhalation (EE) ratio of the patient, and wherein modulating the breathing visualization patch based on the respiratory parameter further comprising modulating intensity of the breathing visualization patch based on the respiratory parameter.

[0084] Example 17. A physical article of manufacture including one or more tangible computer-readable storage media encoding computer-executable instructions for executing on a computer system a computer process to determine respiratory rate of a patient, thecomputer process comprising: determining, based on an image signal received from a camera focused on at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient; communicating the respiratory parameter to a display apparatus; modulating a breathing visualization patch based on the respiratory parameter; and displaying the modulated breathing visualization patch.

[0085] Example 18. The physical article of manufacture of Example 17, wherein the computer process comprising: determining, based on the image signal received from the camera focused on at least a portion of a patient a respiratory waveform of the patient; and communicating the respiratory waveform of the patient to the display apparatus.

[0086] Example 19. The physical article of manufacture of Example 18, wherein the computer process further comprising: modulating brightness of a breathing visualization patch based on an amplitude of a respiratory waveform of the patient; and displaying the breathing visualization patch on a representation of torso of the patient.

[0087] Example 20. The physical article of manufacture of Example 17, wherein the respiratory parameter is an inhalation: exhalation (EE) ratio of the patient and RR of the patient, and wherein modulating the breathing visualization patch based on the respiratory parameter further comprising modulating intensity of the breathing visualization patch based on the respiratory parameter.

Claims

ClaimsWHAT IS CLAIMED IS:

1. A method, comprising: determining, based on an image signal received from a camera in view of at least a portion of a patient, a respiratory parameter representing a breathing rate of the patient; communicating the respiratory parameter to a display; modulating a breathing visualization patch based on the respiratory parameter; and displaying the modulated breathing visualization patch on the display.

2. The method of claim 1, wherein modulating the breathing visualization patch comprises modulating at the breathing rate.

3. The method of claim 1 or 2, wherein displaying the breathing visualization patch comprises displaying on a representation of a torso of the patient.

4. The method of any preceding claim, further comprising: determining, based on the image signal received from the camera, a respiratory waveform of the patient; and communicating the respiratory waveform of the patient to the display.

5. The method of claim 4, wherein modulating the breathing visualization patch further comprises modulating a brightness of the breathing visualization patch based on an amplitude of the respiratory waveform of the patient.

6. The method of claim 4, wherein modulating the breathing visualization patch further comprises modulating a color of the breathing visualization patch based on an amplitude of the respiratory waveform of the patient.

7. The method of claim 4, wherein modulating the breathing visualization patch further comprises modulating at least one of a color and a brightness of the breathing visualization patch based on a phase of the respiratory waveform of the patient.

8. The method of claims 1-5, wherein the respiratory parameter comprises an inhalation exhalation (I:E) ratio of the patient, and wherein modulating the breathing visualization patch based on the respiratory parameter further comprising modulating a color of the breathing visualization patch based on the I:E ratio.

9. The method of claim 1, further comprising intermittently communicating the image signal to the display.

10. A system comprising : a camera in view of a patient; memory; one or more processor units in communication with the camera and the memory; a breathing visualization system stored in the memory and executable by the one or more processor units, the breathing visualization system encoding computer-executable instructions on the memory for executing on the one or more processor units a computer process, the computer process comprising: determining, based on an image signal received from the camera, a respiratory parameter representing a breathing rate of the patient; communicating the respiratory parameter to a display; modulating a breathing visualization patch based on the respiratory parameter; and displaying the modulated breathing visualization patch on the display.

11. The system of claim 10, wherein the camera comprises a depth-sensing camera.

12. The system of claim 10 or 11, wherein communicating the respiratory parameter to the display comprises communicating the respiratory parameter intermittently.

13. The system of claim 10-12, wherein displaying the modulated breathing visualization patch comprises superimposing the modulated breathing visualization patch on an image of the patient.

14. The system of claim 13, wherein the image comprises an image from the image signal.

15. The system of claim 10-14, wherein modulating the breathing visualization patch comprises varying a size of the breathing visualization patch based on the respiratory parameter.

Images