Sensor alignment for non-contact patient monitoring system

Abstract

The disclosed techniques enable alignment of a sensor with respect to a patient's environment by determining a breathing region of a patient; detecting sensor data by the sensor, the sensor data including a plurality of distances between a position of the sensor and the breathing region; generating an aggregated alignment metric based at least in part on the detected sensor data; determining that the aggregation alignment metric satisfies a dislocation condition; classifying a misalignment characteristic of the sensor relative to the determined breathing region based at least in part on satisfying the misalignment condition; and generating an instruction to provide a misalignment notification based at least in part on the classified misalignment characteristic.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 367,180, filed on June 28, 2022, entitled “SENSOR ALIGNMENT FOR A NON-CONTACT PATIENT MONITORING SYSTEM,” the entire contents of which are hereby incorporated by reference. Technical Field

[0003] The present disclosure relates to information display for non-contact patient monitoring, and more specifically, to information display that informs a user about the alignment of a sensor relative to a patient environment. Information can be calculated based on depth measurements and / or captured images made by a non-contact patient monitoring system (including a depth sensing camera), and instructions to display alignment information can be derived from the information, the alignment information representing the alignment of the sensor relative to the patient environment. The information may include a classification representing misalignment characteristics of the sensor relative to the patient environment. The information can be derived relative to a region of interest (hereinafter referred to as a "ROI") detected by the non-contact patient monitoring system, the ROI including a portion of the patient environment. The non-contact patient monitoring system can provide instructions to display the alignment information on a display device or sensor. Background Art

[0004] Depth sensing technologies have been developed that, when integrated into contactless patient monitoring systems, can be used to determine a number of physiological and environmental parameters, such as respiratory rate, tidal volume, minute ventilation, etc. Such parameters can be displayed on a display, providing a basic visualization of these parameters to the clinician. For example, a respiratory rate measurement can be presented.

[0005] However, the validity of physiological and environmental parameters depends on the alignment of the sensor (e.g., depth sensing camera) relative to the ROI in the patient's environment. If the sensor is misaligned relative to the ROI, then the physiological and environmental parameters derived from the sensor measurements are likely to be incorrect and may not be useful. Summary of the invention

[0006] In some embodiments, the disclosed technology qualifies alignment of a sensor relative to a patient environment by: determining a breathing zone of the patient; detecting sensor data by the sensor, the sensor data comprising a plurality of distances between a location of the sensor and the breathing zone; generating an aggregate alignment metric based at least in part on the detected sensor data; determining that the aggregate alignment metric satisfies a misalignment condition; classifying a misalignment characteristic of the sensor relative to the determined breathing zone based at least in part on satisfying the misalignment condition; and generating instructions to provide a misalignment notification based at least in part on the classified misalignment characteristic.

[0007] In some embodiments, the disclosed technology qualifies alignment of a sensor relative to a patient environment by: determining a respiratory region of the patient; detecting sensor data by the sensor, the sensor data comprising one or more of: a captured image of the respiratory region; and a plurality of distances between a location of the sensor and the respiratory region; determining a sensed quality of the determined respiratory region based on the detected sensor data by a predefined relationship between the sensor data and the sensed quality of the respiratory region; determining whether the sensed quality of the determined respiratory region satisfies a misalignment condition; classifying a misalignment characteristic of the sensor relative to the determined respiratory region based at least in part on satisfying the misalignment condition; and generating an instruction to provide a misalignment notification based at least in part on the classified misalignment characteristic.

[0008] In some embodiments, the disclosed technology provides a system for qualifying the alignment of a sensor relative to a patient environment. The system includes one or more hardware processors, a sensor operable to detect sensor data (including a plurality of distances between a location of the sensor and a determined respiratory zone of a patient), and an alignment manager executable by the one or more hardware processors. The alignment manager is configured to generate an aggregate alignment metric based at least in part on the detected sensor data, determine that the aggregate alignment metric satisfies a misalignment condition, classify a misalignment characteristic of the sensor relative to the determined respiratory zone based at least in part on satisfying the misalignment condition, and generate instructions to provide a misalignment notification based at least in part on the classified misalignment characteristic.

[0009] In some embodiments, the disclosed technology provides a system for qualifying the alignment of a sensor relative to a patient environment. The system includes one or more hardware processors, a sensor operable to detect sensor data, the sensor data including a captured image of a determined respiratory region of the patient and one or more of a plurality of distances between a position of the sensor and the determined respiratory region. The system further includes an alignment manager executable by the one or more hardware processors. The alignment manager is configured to determine the sensed quality of the determined respiratory region based on the detected sensor data through a predefined relationship between the sensor data and the sensed quality of the respiratory region, determine whether the sensed quality of the determined respiratory region satisfies a misalignment condition, classify a misalignment characteristic of the sensor relative to the determined respiratory region based at least in part on satisfying the misalignment condition, and generate instructions to provide a misalignment notification based at least in part on the classified misalignment characteristic. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Many aspects of the present disclosure may be better understood with reference to the following drawings. The components in the drawings are not necessarily drawn to scale. Instead, emphasis is placed on clearly demonstrating the principles of the present disclosure. The drawings should not be considered to limit the present disclosure to the specific embodiments depicted, but are for explanation and understanding only.

[0011] Figure 1 is a schematic diagram of an implementation of a video-based patient monitoring system configured in accordance with various embodiments of the present technology.

[0012] Figure 2 is a block diagram illustrating an implementation of a video-based patient monitoring system having a computing device, a server, and one or more image capture devices and configured in accordance with various embodiments of the present technology.

[0013] Figure 3 is a display view of an implementation of a user interface for a video-based patient monitoring system configured in accordance with various embodiments of the present technology.

[0014] Figure 4 is a top view of an embodiment of alignment of a patient relative to a sensor.

[0015] Figure 5 are display views of embodiments of an alignment indicator system that may be presented on a sensor device or a display.

[0016] Figure 6 is a flow chart of an implementation of a method for qualifying alignment of a sensor relative to a patient environment, configured in accordance with various embodiments of the present technology. DETAILED DESCRIPTION

[0017] The present disclosure relates to information display for contactless patient monitoring. The technology described herein can be incorporated into the system and method for contactless patient monitoring. As described in more detail below, the described technology can include obtaining respiratory data (such as using an image sensor (e.g., a depth sensing camera) via contactless patient monitoring) and displaying respiratory data. The technology can further include capturing image data of a predefined portion of a patient and / or distance data generated relative to a predefined portion or ROI of the patient to determine whether and / or how the sensor is misaligned relative to the patient or ROI and / or classify it. The technology can further provide instructions to the display indicating misalignment and / or potential corrective actions in response to misalignment. The instruction can include displaying an indication of misalignment or misalignment classification on a display or an indicator physically located on the sensor.

[0018] References Figures 1 to 6 Specific details of several embodiments of the technology of the present invention are described. Although many embodiments are described with respect to devices, systems and methods for image-based monitoring of the respiration of human patients and the related display of such monitoring, other applications and other embodiments other than those described herein are also within the scope of the technology of the present invention. For example, at least some embodiments of the technology of the present invention can be used for image-based monitoring of the respiration of other animals and / or non-patients (e.g., elderly or newborn individuals at home). It should be noted that other embodiments other than the embodiments disclosed herein are also within the scope of the technology of the present invention. Further, embodiments of the technology of the present invention may have different configurations, components and / or programs than those shown or described herein. In addition, those of ordinary skill in the art will understand that embodiments of the technology of the present invention may have configurations, components and / or programs other than those shown or described herein, and without departing from the technology of the present invention, these and other embodiments may not have some of the configurations, components and / or programs shown or described herein.

[0019] Figure 11 is a schematic diagram of an embodiment of a patient 112 and a video-based patient monitoring system 100 configured according to various embodiments of the present technology. The system 100 includes a non-contact detector 110 and a computing device 115. In some embodiments, the non-contact detector 110 may include one or more image capture devices, such as one or more cameras. In the illustrated embodiment, the non-contact detector 110 includes a camera 114 (which may be a camera suitable for capturing video or other time-series image data). The non-contact detector 110 of the system 100 is placed away from the patient 112. More specifically, the sensor or camera 114 of the non-contact detector 110 is positioned away from the patient 112, that is, the sensor or camera is spaced apart from the patient 112 and does not contact the patient. The camera 114 includes a detector exposed in a field of view (FOV) 116 covering at least a portion of the patient 112. In an embodiment, the camera 114 is operable to detect electromagnetic energy or other energy of any spectrum (e.g., infrared, visible light, thermal energy, x-rays, microwaves, radio, gamma rays, etc.).

[0020] The camera 114 can capture a series of images over time. The camera 114 can be a depth sensing camera, such as a Kinect camera from Microsoft Corp. (Redmond, Washington) or an Intel camera, such as the D415, D435, and SR305 cameras from Intel Corp. (Santa Clara, California). The depth sensing camera can detect the distance between the camera and objects within its field of view. This information can be used to determine that the patient 112 is within the FOV 116 of the camera 114 and / or to determine one or more regions of interest (ROIs) on the patient 112 to be monitored. Once the ROI 102 is identified, the ROI 102 can be monitored over time, and changes in the depth of the area within the ROI 102 and / or the captured images can indicate movement of the patient 112 associated with breathing.

[0021] As described in more detail in U.S. Patent Application Publication No. 2019 / 0209046, these movements or regional changes within ROI 102 can be used in addition or alternatively to determine various respiratory parameters, such as tidal volume, minute ventilation, respiratory rate, respiratory volume, etc. As described in more detail in U.S. Patent Application Publication No. 2020 / 0046302, these movements or regional changes within ROI 102 can also be used to detect various respiratory abnormalities. Various respiratory abnormalities can include, for example, low flow, apnea, rapid breathing (shortness of breath), slow breathing, intermittent or irregular breathing, shallow breathing, obstructed breathing and / or impaired breathing, etc. U.S. Patent Application Publication Nos. 2019 / 0209046 and 2020 / 0046302 are incorporated herein by reference in their entirety.

[0022] In some embodiments, system 100 determines a skeleton-like outline of patient 112 to identify one or more points from which to infer an ROI. For example, the skeleton-like outline can be used to find a chest center point, a shoulder point, a waist point, and / or any other point on the body of patient 112. These points can be used to determine one or more ROIs. For example, Figure 1 As shown, the ROI 102 can be defined by filling in the area around the point 103 (such as the center point of the chest). Certain determined points can define the outer edge of the ROI 102, such as the shoulder point. In other embodiments, instead of using a skeleton, other points are used to determine the ROI. For example, a face can be recognized and a chest area that is proportional and spatially related to the face can be inferred. In other embodiments, reference points of the patient's chest can be obtained (for example, through a previous 3-D scan of the patient), and then the reference points are aligned with the patient's current 3-D scan. In these and other embodiments, the system 100 can use a portion of the patient 112 within a certain depth range from the camera 114 to define the ROI around a certain point. In other words, once the system 100 determines the point from which the ROI is inferred, the system 100 can fill the ROI with depth information from the camera 114 (which can be a depth sensing camera). For example, if point 103 on the chest is selected, the portion of the patient 112 surrounding point 103 (which is similar to the depth or distance of point 103 from camera 114 ) is used to determine ROI 102 .

[0023] In another example, the patient 112 may wear specially configured clothing or be covered with specially configured bedding or other materials (not shown) that respond to visible light or respond to electromagnetic energy of different spectra, which include one or more features for indicating points on the patient 112 body (such as the patient's shoulder and / or the center of the patient's chest). The one or more features may include a visually encoded message (e.g., a barcode, a QR code, etc.) and / or a brightly colored shape that contrasts with the rest of the patient's clothing. In these and other embodiments, the one or more features may include one or more sensors that are operable to indicate their location by transmitting light or other information to the camera 114. In these and other embodiments, the one or more features may include a grid or another recognizable pattern to help the system 100 identify the patient 112 and / or the patient's movement. In some embodiments, one or more features may be attached to clothing using a fastening mechanism (e.g., an adhesive, a pin, etc.). For example, a small sticker may be placed on the patient's shoulder and / or the center of the patient's chest, which may be easily identified in an image captured by the camera 114. The system 100 may identify one or more features on the patient's clothing to determine a specific point on the patient's 112 body. The system 100 may, in turn, use these points to identify the patient 112 and / or define the ROI 102 .

[0024] In some embodiments, the system 100 may receive user input to identify a starting point for defining the ROI 102. For example, an image may be reproduced on a display 122 of the system 100, allowing a user of the system 100 to select a patient 112 for monitoring (which may be helpful in situations where there are multiple objects within the FOV 116 of the camera 114) and / or allowing the user to select a point on the patient 112 from which an ROI may be determined (e.g., point 103 on the chest of the patient 112). In other embodiments, other methods may be used to identify the patient 112, identify a point on the patient 112, and / or define one or more ROIs.

[0025] In an embodiment, the ROI 102 may include a respiratory region of the patient 112. The respiratory region is a predefined region of the patient 112 (e.g., chest, face, nostrils, etc.) whose movement is attributable to breathing. For example, the chest of the patient 112 typically moves during breathing. The movement of the chest cavity causes the lungs to expand. Respiratory movement is typically manifested as a continuous alternation of chest expansion and relaxation. Other ROIs may include the patient's hands, face, or other anatomical features. The respiratory region can be displayed on the display 122 as a region indicator or "mask" on an image of the patient 112 and / or the patient's environment. The respiratory region and region indicator can be determined as described in U.S. patent application Ser. No. 16 / 713,265, which is incorporated herein by reference.

[0026] Detection of ROI 102 can help determine whether camera 114 (or another sensor) is properly aligned with the patient environment. The type of misalignment between camera 114 (or another sensor) and other parts of ROI or patient environment can include one or more of misalignment direction, misalignment size, rotational misalignment, translational misalignment, tilt angle misalignment and distance misalignment. Rotational misalignment can include azimuth misalignment of camera 114 or other sensor, and azimuth correction can be used to correct alignment. In an embodiment, system 100 can be configured to detect rotational misalignment by detecting and determining relative orientation or two or more of patient face, patient chest, patient hand, visual indicator representing all or part of ROI 102, and elements covering anatomical features of the patient (e.g., blanket, mask or clothing). Rotational misalignment can also be based on the relative orientation of the longitudinal length of ROI (e.g., the central longitudinal axis of ROI 102 should substantially bisect FOV). Translational misalignment is a misalignment in which camera 114 is incorrectly translated relative to the patient environment, so translational alignment correction can correct such misalignment. For example, a correct translational alignment may include the center of the ROI 102 being at the center of the FOV of the camera 114 (e.g., when the lens or detector of the camera 114 is positioned substantially orthogonally relative to the captured surface (e.g., the ROI 102, the bed on which the patient lies, or the floor in the patient environment)). Tilt misalignment is the angular alignment of the lens of the camera 114 (or other sensor detection element such as a depth sensor) relative to the ROI 102, which may be corrected by tilt correction. In embodiments, the aligned tilt angle may be a zero angle, which indicates that the lens or detection element detects light or electromagnetic radiation substantially (e.g., on average) orthogonally relative to a portion of the patient environment (such as the ROI 102), or that the lens itself is substantially (e.g., on average) parallel to the surface of the ROI 102. The misalignment may be a distance misalignment, which causes the distance between the cameras 114 to be outside a predefined distance range. In embodiments, the ROI 102 represents a breathing region of the patient 112 or a portion of the patient environment whose motion indicates breathing. In other embodiments, a respiratory region separate from and / or adjacent to ROI 102 may be determined in a manner similar to the manner in which ROI 102 is determined. For example, embodiments are contemplated in which ROI 102 is a region on the chest region of patient 112 and the determined respiratory region is a region on the face of patient 112 substantially adjacent to or proximate to the chest region.

[0027] The camera 114 may be used to generate sensor data representing one or more of an image of the ROI or a distance between a point in the ROI and the camera 114 to distinguish respiratory motion of the patient 112 that is (at least largely) attributable to respiration from non-respiratory motion of the patient 112 that is attributable to motion other than respiration. The sensor data generated by the camera 114 (including captured images and / or signals representing distances from points within the ROI and the camera 114) may be sent to a computing device 115 via a wired or wireless connection 120. The computing device 115 may include a hardware processor 118 (e.g., a microprocessor), a display 122, and / or a hardware memory 126 for storing software and computer instructions. Sequential image frames of the patient and / or distance signals representing distances from the patient 112 are recorded by the camera 114 and sent to the hardware processor 118 for analysis. The analysis may be performed by a signal processor and / or an alignment manager executable by the hardware processor 118. The analysis may include determining whether the camera 114 or other sensor is aligned relative to the patient environment. In embodiments, the camera 114 includes an indicator that provides an indication of the alignment or misalignment of the camera 114 relative to the patient environment. In some embodiments, the display 122 may additionally or alternatively provide an indication of the alignment or misalignment of the camera 114 as to whether it is aligned or misaligned. The misalignment indication may indicate one or more types of misalignment, one or more directions of misalignment, one or more magnitudes of misalignment (e.g., different magnitudes for different types of misalignment), and / or simply indicate that the camera 114 is misaligned.

[0028] In an embodiment, the camera 114 (or another sensor device) includes an alignment indicator 124. In the illustrated embodiment, the alignment indicator 124 is an indicator light that can emit different lights depending on whether the camera 114 is aligned relative to the patient 112. For example, in an embodiment, the alignment indicator 124 emits a green light when the camera 114 is properly aligned and a red light when the camera 114 is misaligned. There can be more complex light arrangements in the camera's alignment indicator 124, such as lights that specifically indicate one or more of the type, direction, and magnitude of misalignment, lights that indicate corrective actions for one or more of the type, direction, and magnitude of misalignment, and an active pixelated display for displaying one or more of the type, direction, and magnitude of misalignment and corrective actions. Displaying the classification and / or corrective actions on the alignment indicator 124 of the camera 114 can help the user align the camera 114 without having to view a separate display (e.g., display 122) that may be located in a different room. In an embodiment, instructions from the alignment manager can additionally or alternatively cause the misalignment classification and / or corrective actions to be displayed on the display 122.

[0029] Display 122 may be remote from camera 114, such as a video screen located separately from hardware processor 118 and hardware memory 126. Other embodiments of computing device 115 may have Figure 1 In some embodiments, computing device 115 may be a server. In other embodiments, Figure 1 The computing device 115 may be further connected to a server (e.g., Figure 2 1 and discussed in more detail below). The captured images / video may be processed or analyzed by a signal processor at computing device 115 and / or a server to determine various parameters of the patient's respiration (e.g., patient respiratory motion, patient non-respiratory motion, tidal volume, minute ventilation, respiratory rate, etc.). In some embodiments, some or all of the processing may be performed by the camera, such as by a hardware processor integrated into the camera, or when some or all of computing device 115 is integrated into the camera.

[0030] Figure 2 is a video-based patient monitoring system 200 (e.g., Figure 1 1 is a block diagram of an embodiment of a video-based patient monitoring system 100 (shown in FIG. 1 ), which has a computing device 210 (e.g., an embodiment of the computing device 115), a server 225, and one or more image capture devices 285, and is configured according to various embodiments of the present technology. In various embodiments, fewer, additional, and / or different components may be used in the system 200. The computing device 210 includes a hardware processor 215 (e.g., an embodiment of the hardware processor 118) coupled to the memory 205. The hardware processor 215 can store and call data and applications in the memory 205, including applications that process information and send commands / signals according to any method disclosed herein. The hardware processor 215 can also (i) display objects, applications, data, etc. on the interface / display 207, and / or (ii) receive input through the interface / display 207. As shown, the hardware processor 215 is also coupled to the transceiver 220.

[0031] Computing device 210 may communicate with other devices (such as server 225 and / or image capture device(s) 285) via (e.g., wired or wireless) connections 270 and / or 280, respectively. For example, computing device 210 may send information about a patient determined from images captured by image capture device(s) 285 to server 225. Computing device 210 may be Figure 1115 in the computing device 115. Thus, the computing device 210 can be located remotely from the image capture device(s) 285, or it can be located locally and near the image capture device(s) 285 (e.g., in the same room). In various embodiments disclosed herein, the hardware processor 215 of the computing device 210 can perform the steps disclosed herein. In other embodiments, the steps can be performed on the hardware processor 235 of the server 225. The hardware processor 235 of the server 225 is coupled to the memory 230. The hardware processor 235 can store and call data and applications in the memory 230. The hardware processor 235 is also coupled to the transceiver 240. In some embodiments, the hardware processor 235 and subsequently the server 225 can communicate with other devices (such as the computing device 210) via the connection 270.

[0032] In some embodiments, the various steps and methods disclosed herein may be performed by both hardware processors 215 and 235. In some embodiments, certain steps may be performed by hardware processor 215 while other steps are performed by hardware processor 235. In some embodiments, information determined by hardware processor 215 may be sent to server 225 for storage and / or further processing.

[0033] In an embodiment, the image capture device(s) 285 generate sensor data, such as captured images and / or signals representing the distance between the image capture device(s) 285 and at least one point in the ROI. In some embodiments, the image capture device(s) 285 are remote sensing devices, such as depth sensing cameras, as described above with reference to Figure 1 as described.

[0034] In some embodiments, the image capture device(s) 285 may be or include some other type of device(s), such as a proximity sensor or proximity sensor array, a thermal sensor or infrared sensor / camera, a sound / acoustic wave or radio wave transmitter / detector, or other device that includes a field of view and can be used to monitor the location and / or characteristics of a patient or a region of interest (ROI) on the patient. Human body imaging techniques may also be utilized in accordance with the methods disclosed herein. For example, a patient may be scanned using backscattered x-ray or millimeter wave scanning techniques, which may be used to define and / or monitor the ROI. Advantageously, such techniques are able to penetrate (e.g., "see") clothing, bedding, or other materials while accurately representing the patient's skin. This allows for more accurate measurements, particularly when the patient is wearing loose clothing or under bedding. The image capture device(s) 285 may be described as local devices because they are relatively close to the patient such that at least a portion of the patient is within the field of view of the image capture device(s) 285.

[0035] In some embodiments, the image capture device(s) 285 may be adjustable to ensure that the patient is captured in the field of view. For example, the image capture device(s) 285 may be physically movable, may have a changeable orientation (such as by rotation or translation), and / or may be capable of changing focus, zoom, or other capture characteristics to allow the image capture device(s) 285 to adequately capture an image of the patient and / or a ROI of the patient. For example, in various embodiments, the image capture device(s) 285 may focus on the ROI, zoom in on the ROI, center the ROI in the field of view by moving the image capture device(s) 285, or otherwise adjust the field of view to allow for better and / or more accurate tracking / measurement of the ROI. The system 200 may include an automatic actuator for aligning the image capture device(s) 285 based on a determination that the image capture device(s) are misaligned. Corrective measures may include one or more of distance correction (between the image capture device(s) and the ROI), azimuth (rotation) correction, translation correction, and tilt correction. In embodiments, the indicator light may also function as a button to drive the actuator to correct the misalignment by taking the indicated type of corrective action to correct the type, direction and / or magnitude of the misalignment.

[0036] In an embodiment, the generated sensor data may include time series data. For example, the sensor data may be arranged in chronological order, possibly with an associated timestamp representing the time of data capture and / or generation. The time series data may represent the movement of the patient over time. The time series data may represent video data of captured images, and may represent the change in distance between (multiple) image capture devices 285 and points in the ROI of the distance signal data. The time series data may be analyzed to show the misalignment over time and / or the change in misalignment over time, and the time series data may be used to determine the misalignment. When analyzing time series data, a duration and / or sample size including one or more time windows (e.g., between a first time and a second time) may be used during the analysis. These durations may be dynamically determined based on the breathing pattern of a particular patient, or may be standardized.

[0037] The system 200 may include an alignment manager for processing sensor data generated by the image capture device(s) 285 and determining whether the image capture device(s) 285 are properly aligned relative to the patient or an element of the patient's environment (e.g., an ROI, a respiratory zone, or an identifiable element near the ROI). The alignment manager may include a hardware element of one or more of the computing device 210, the image capture device(s) 285, and the server 225, may include a software element executable by one or more of the processor(s) of the image capture device(s) 285, the hardware processor 215, and the hardware processor 235, or may include a hybrid system of software and hardware included in one or more of the computing device 210, the image capture device(s) 285, and the server 225.

[0038] In some embodiments, the alignment manager generates an aggregate alignment metric of the alignment between the image capture device(s) 285 and the patient environment. In these embodiments, the alignment manager receives generated sensor data (e.g., captured images and / or sensed distances of elements of the patient environment (e.g., a patient's respiratory zone)) and is operable to generate an aggregate alignment metric based at least in part on the detected sensor data. The aggregate alignment metric may include an aggregate distance or angle metric based on one or more of a mean, a median, and a trimmed mean of the detected distances between the image capture device(s) 285 and the elements of the patient environment. The aggregate alignment metric may consider multi-dimensional misalignment of different misalignment types (e.g., one or more of rotational misalignment, translational misalignment, tilt angle misalignment, and distance misalignment). The aggregate alignment metric may further consider the magnitude and / or direction of the misalignment type. The alignment manager may then determine whether the aggregate alignment metric satisfies a misalignment condition. The misalignment condition may include a predefined threshold and / or a predefined value range representing the magnitude and / or direction of the misalignment type. In an embodiment, the aggregate alignment metric may include a score based on the value of the misalignment type. If the score is multifactorial (eg, based on more than one alignment type or based on different methods of determining the nature and extent of the alignment), the score may comprise a weighted average of each of the different factors.

[0039] An embodiment of the aggregate alignment metric may include an alignment score. In one embodiment, the alignment score is determined based on an aggregate distance between the image capture device(s) 285 and the determined breathing region. For example, the alignment manager can determine the alignment score as in Eq. 1.

[0040] Alignment score = | distance agg -1| (1)

[0041] Aggregation distance agg) may include one or more of an average, a median, and a trimmed mean of the sensed distance data. In equation one, the absolute value is the aggregate distance minus one meter. If the distance differs too much from one meter, the alignment score will reflect that the (multiple) image capture devices 285 are misaligned. Predefined distances other than one meter are considered and may be substituted into one or more of equations 1 and 2. In an embodiment, the alignment score will be further based on the aggregate angle, as presented in equation 2.

[0042] Alignment score = | distance agg -1|+|angle agg | (2)

[0043] Convergence angle agg ) may be based on one or more of an average, median, or mean of angles detected between the image capture device(s) 285 and the determined respiratory zone, or an angle representing a mean or median vector of distances determined between the image capture device(s) 285 and elements of the patient environment. The misalignment condition may be based on one or more alignment scores. In embodiments, the misalignment condition may include that the misalignment condition is satisfied when the alignment score exceeds a threshold alignment score or falls outside an acceptable alignment score range.

[0044] In embodiments, the alignment manager additionally or alternatively determines a sensed quality of the respiratory region. In these embodiments, the alignment manager receives detected sensor data (e.g., a captured image of an element of a patient's environment (such as a patient's respiratory region) and / or a sensed distance to the element of the patient's environment) and is configured to determine the sensed quality of the determined respiratory region based on the detected sensor data through a predefined relationship between the sensor data and the sensed quality of the respiratory region. The sensed quality may include one or more of: a detected size or shape of the determined respiratory region, a detected fill ratio of an image representing the respiratory region (e.g., the extent to which there are gaps in a visual indicator overlaid on the image representing the respiratory region), and a detected shape or size of a respiratory mask (e.g., the region determined to be the patient's respiratory region). The alignment manager may then determine whether the sensed quality satisfies a misalignment condition. For example, the sensed shape or size of the sensed portion of the determined respiratory region may vary depending on the angle and / or distance at which the image capture device 285 senses the respiratory region. If the sensed shape or size of the respiratory region is consistent with the predefined shape and / or size parameters of the misalignment condition, the sensed quality of the sensed shape or size may satisfy the misalignment condition.

[0045] In embodiments where the sensed size of a detected portion of the respiratory region is used to determine misalignment, the sensed size of the respiratory region relative to the size of a comparison region (e.g., one or more of the ROI, the entire respiratory region, an overlay target portion assigned to sensor data by a sensor or image processor when the sensor data was captured) may indicate the distance between the image capture device(s) 285 and the respiratory region. The ratio may be determined based on the number of pixels in the respiratory region relative to the number of pixels in the rest of the perceived environment (or within a predefined range of the field of view of the image capture device(s) 285), or the relative size may be determined by integrating depth or distance in the respiratory region or another indicative region (e.g., a region covering one or more of the patient's chest, head, arms, hands, other anatomical features that may demonstrate the orientation of the image capture device(s) 285 relative to the patient) to generate a physical indicator of the size of the indicative region.

[0046] In embodiments where the sensed shape is a chest in the patient's breathing region, it may be expected that the sensed shape is substantially elliptical and substantially symmetrical about a bisector of the elliptical shape. Dissimilarity in size and / or shape may be used (e.g., as an element of a misalignment condition) to determine misalignment of the image capture device(s) 285.

[0047] Additionally or alternatively, if a sensed fill ratio (a ratio of a distance from a detected location in the determined breathing area where the lighting condition is not satisfied and thus cannot be presented as light in the breathing area or mask to an absolute distance from the location in the determined breathing area or relative to other detected distances that satisfy the lighting condition in the determined breathing area) is an element of a misalignment condition, the sensed quality of the fill ratio may satisfy the misalignment condition if the ratio falls within a predetermined range or satisfies a predetermined threshold for the misalignment condition.

[0048] In an embodiment, the predefined relationship between the generated sensor data and the sensing quality used to determine that the misalignment condition is satisfied can be represented by an inference model (such as a machine learning model). The inference model can be pre-trained (e.g., before making an inference determination) based on the generated labeled sensor data that is labeled to represent the sensing quality of the breathing area. For example, the inference model can be pre-trained by inputting time series data of images captured at different times (e.g., a first time and a second time) and / or generated distance signals, which are labeled with the labeled sensing quality of the determined breathing area. The inference model trainer can then compare the output of the inference model with the label associated with the input and determine the loss between the output and the associated label. The inference model trainer can then back-propagate or otherwise distribute the loss through the inference model (e.g., by adjusting one or more of the weights, activation functions, and biases represented in one or more nodes and edges of the neural network of the machine learning model). By repeating this process with different generated labeled sensor data, the inference model can represent the predefined relationship between the generated sensor data and the sensing quality.

[0049] In embodiments, the labeling and output of the inference model may include data (e.g., probabilities and / or statistics) representing a confidence level in the output determination. In embodiments, a predefined relationship including an inference model may be similarly applied between detected sensor data and one or more aggregated alignment metrics to determine aggregated alignment metrics based on the detected sensor data, which aggregated alignment metrics may be similarly pre-trained using sensor data labeled with the labeled aggregated alignment metrics. Further, embodiments are contemplated in which a single inference relationship may determine a classification or score for misalignment based on the detected sensor data, one or more sensed qualities, and one or more aggregated alignment metrics representing alignment between (multiple) image capture devices and a determined respiratory region.

[0050] In an embodiment, the inference model may include, but is not limited to, one or more of the following: data mining algorithm, artificial intelligence algorithm, masked learning model, natural language processing model, neural network, artificial neural network, perceptron, feedforward network, radial basis neural network, deep feedforward neural network, recurrent neural network, long / short term memory network, gated recurrent neural network, autoencoder, variational autoencoder, denoising autoencoder, sparse autoencoder, Bayesian network, regression model, decision tree, Markov chain, Hopfield network, Boltzmann machine, restricted Boltzmann machine, deep belief network, deep convolutional network, genetic algorithm, deconvolution neural network, deep convolutional inverse graph network, generative adversarial network, liquid machine, extreme learning machine, echo state network, deep residual network, Kohonen network, support vector machine, federated learning model and neural Turing machine.

[0051] In an embodiment, the inference model can be trained by the inference model trainer using a training method. Examples of training methods (e.g., inference and / or machine learning methods) may include, but are not limited to, one or more of the following: masked learning modeling, unsupervised learning, supervised learning, reinforcement learning, self-learning, feature learning, sparse dictionary learning, anomaly detection, robotic learning, association rule learning, manifold learning, dimensionality reduction, bidirectional transformation, unidirectional transformation, gradient descent, autoregression, autoencoding, permutation language modeling, dual-stream self-attenuation, federated learning, etc.

[0052] The predefined relationships may be based on the patient's demographics. In an embodiment, the predefined relationships are established specifically for the patient's demographics by training the inference model exclusively or primarily on labeled data associated with the demographics. In another embodiment, the inference model is configured and / or pretrained to take demographic data as input, and the inference relationships within the inference model internally consider the demographic data.

[0053] The system may additionally or alternatively determine other respiratory measurements, such as values ​​of various parameters (e.g., patient respiratory motion, patient non-respiratory motion, tidal volume, minute ventilation, respiratory rate, etc.). In an embodiment, the signal processor may determine the respiratory measurements based on the generated sensor data. In other embodiments, the signal processor may receive respiratory measurements generated by other means, including one or more of transthoracic impedance measurement, electrocardiogram, carbon dioxide analyzer, spirometer, pulse oximeter, and manual user input. Embodiments in which the signal processor does not process respiratory measurements other than patient motion classification are also contemplated. In an embodiment, the respiratory measurements may be considered in determining one or more of the aggregate alignment metric, determining the sensing quality, and the misalignment condition. For example, the misalignment condition may be based on the periodicity of the respiratory volume waveform. Irregular periodicity may indicate misalignment, while regular waveform periodicity may indicate correct alignment. Similarly, if the respiratory rate measurements and / or associated confidence values ​​of the respiratory rate measurements differ significantly from expected or predefined respiratory rate measurements (e.g., satisfy a dissimilarity condition), this may indicate that the image capture device(s) 285 are misaligned relative to the patient environment.

[0054] After determining one or more alignment qualities (e.g., aggregate alignment metrics and / or sensing qualities), the alignment manager may determine whether the one or more alignment qualities satisfy one or more misalignment conditions based on the one or more alignment qualities. In an embodiment, the misalignment condition may include a predefined threshold and / or a predefined value range representing the magnitude and / or direction of the misalignment type. In an embodiment, the aggregate alignment metric may include a score based on the value of the misalignment type. If the score is multi-factorial (e.g., based on values ​​of more than one alignment type or based on different methods of determining the nature and degree of alignment), the score may include a weighted average or geometric mean of each different factor.

[0055] In embodiments, the misalignment condition may be based at least in part on the relative orientation of the region near the ROI. For example, the determined respiratory region may include the thoracic region and the ROI of one or more anatomical features (including but not limited to the determined ROIs adjacent to the head, arms, legs, or hands). If the head is not in the proper position directly above the torso, it may indicate that the image capture device(s) 285 is misaligned in rotation (azimuth misalignment).

[0056] In other embodiments, the misalignment condition may include hard cutoff values ​​for certain parameters. For example, even if the determination that one or more misalignment conditions are met is based on many factors, one or more of the factors may have a threshold or range where the misalignment condition will automatically fail. For example, the misalignment condition may include a maximum convergence distance of 1.5 meters between the image capture device(s) 285 and the breathing zone. In this example, even if all other values ​​indicate that the image capture device(s) 285 are aligned, the misalignment condition may be met based solely on the determination that the convergence distance is 1.6 meters, regardless of all other factors. In this embodiment, detecting that a determined breathing zone distance is greater than 1.5 meters indicates that a misalignment has occurred, regardless of other considerations.

[0057] In embodiments where the alignment manager determines one or more of an aggregate alignment metric, sensing quality, satisfaction of a misalignment condition (or a misalignment condition that the alignment manager uses to determine satisfaction), and a misalignment classification based on more than one of the above-described factors, these factors may be weighted and considered in a weighted sum, or these factors may be used to determine a geometric mean as an overall misalignment score or a score for a specific misalignment element (e.g., misalignment type, direction, and / or magnitude).

[0058] In embodiments, the alignment manager may classify one or more alignment qualities. In these embodiments, the alignment manager may include a misalignment classifier operable to classify misalignment characteristics based on satisfying a misalignment condition. The classified misalignment characteristics may include one or more of: misalignment of the image capture device(s) 285 relative to the determined breathing region, one or more types of misalignment (e.g., rotation, translation, tilt angle, and / or distance), and one or more (e.g., one or more types of) misalignment magnitudes, etc. Embodiments are contemplated in which the classification is performed by an inferential relationship that is the same as or different from an inferential relationship that determines one or more of the aggregate alignment metric and / or sensing qualities of the determined breathing region. In embodiments using the same inferential relationship, the operation of determining the sensing quality and / or aggregate metric and the operation of classifying the misalignment characteristics may be combined into a single operation (e.g., the input is sensor data and the output is a misalignment classification).

[0059] In an embodiment, the alignment manager or instruction generator generates instructions to provide a misalignment notification based at least in part on satisfying a misalignment condition and / or a classification of the misalignment (e.g., determined based on satisfying the misalignment condition). The alignment manager and / or the instruction generator for the alignment manager, executable by a hardware processor of the system 200 (e.g., one or more of the hardware processors 215, 235, or the hardware processors of the image capture device(s) 285), may generate instructions for displaying data generated by the alignment manager. The instructions may include data representing instructions to display the misalignment notification. The misalignment notification may include one or more of: an indication that the image capture device(s) 285 are misaligned with the determined respiratory region, a type of misalignment, a magnitude of the misalignment, and one or more corrective actions to be taken to align the image capture device(s) relative to the determined respiratory region.

[0060] In embodiments, the instructions and / or indications may include data representing one or more of: a motion classification flag, a classification-specific display, an overlay display (e.g., configured to overlay a displayed element in a user interface to indicate misalignment relative to a determined respiratory zone), an image representation of patient motion (e.g., a captured image and / or a visual or video representation of a generated distance signal), an audio signal, a flashing element (e.g., wherein the amount of light of an element of a display alternately increases and decreases), different alarms, codes representing the above items, and the like. In embodiments where the classification is one of a plurality of classifications, each classification may correspond to a different display. The signal processor may output data representing the motion classification or a specific display associated with the alignment classification in a memory of the system 200. For example, different classifications may be represented in a display by different colors, different amounts of light in a display, different flashing frequencies in a display, and the like.

[0061] In embodiments where the misalignment classification represents different degrees of misalignment or confidence in its measurement, the corresponding display may represent a spectrum or range of colors, amounts of light, or flickering frequencies based on the magnitude of one or more of the different degrees of misalignment type and / or different degrees of confidence in determining the misalignment. In one embodiment, the instructions include activating one or more lights on the image capture device(s) 285 (e.g., in the illustrated embodiment, Figure 1 The light display may be simple and indicate the presence of misalignment by emitting a different color of light, activating a light only when the image capture device(s) 285 are in one of alignment or misalignment, flashing a light only when the image capture device(s) 285 are in one of alignment or misalignment, etc. The display may be more complex in that it indicates one or more of the type and magnitude of the misalignment and the type and magnitude of corrective action to correct the misalignment. See for examples of different indicator lights. Figure 5Alignment indicator system 500. In embodiments, the magnitude of any particular type of misalignment may be represented by a frequency, color, and / or brightness of lights or pixels that represent a particular type or direction of misalignment or a particular type or direction of corrective action to correct the misalignment.

[0062] In an embodiment, the instructions for providing a misalignment notification may be configured to cause the display to display the misalignment notification as an element superimposed on or below (e.g., displayed behind) another displayed respiratory measurement result or image of the patient, including an ROI image for indicating the alignment of the (multiple) image capture device 285. For example, the display may be configured to display a measured respiratory rate, and may display an indication of a misalignment classification that exceeds or falls below the displayed respiratory rate. The superimposed or underlay display of the misalignment notification may be configured to form a visual contrast with the displayed respiratory measurement result. For example, these elements may have different colors and / or different transparencies, and / or the displayed misalignment notification may be at least partially transparent to maintain visibility of the displayed respiratory measurement result (e.g., appearing as a highlight or patch of the displayed respiratory measurement result). Additionally or alternatively, the displayed misalignment notification may be superimposed on the ROI image. In an alternative embodiment, when the alignment manager determines that the (multiple) image capture device 285 is misaligned, the instructions indicate that the display shall not display the respiratory measurement result and / or the patient image.

[0063] In embodiments, the misalignment manager may be communicatively coupled to a patient presence detector (e.g., hardware or software executable by one or more hardware processors) that detects the presence of a patient. In embodiments where the presence detector determines that a patient is not present, the misalignment manager may do one or more of the following: deactivate itself (e.g., until the presence of a patient is detected), deactivate any display or indication that the misalignment manager would instruct to display a misalignment indication, and transmit instructions to display a different alignment indication (e.g., color, light frequency, and / or intensity) that reflects the absence of a patient.

[0064] The device shown in the illustrative embodiment can be used in various ways. For example, any one of the connections 270 and 280 can be changed. Any one of the connections 270 and 280 can be a hard-wired connection. The hard-wired connection can involve connecting the device through a USB (universal serial bus) port, a serial port, a parallel port, or another type of wired connection that can facilitate the data and information transmission between the processor of the device and the second processor of the second device. In another embodiment, any one of the connections 270 and 280 can be a docking piece that a device can be inserted into another device. In other embodiments, any one of the connections 270 and 280 can be a wireless connection. These connections can take the form of any kind of wireless connection, including but not limited to Bluetooth connection, Wi-Fi connection, infrared, visible light, radio frequency (RF) signal or other wireless protocols / methods. For example, other possible modes of wireless communication can include near-field communication, such as passive radio frequency identification (RFID) and active RFID technology. RFID and similar near-field communication can enable various devices to communicate within a short distance when placed close to each other. In yet another embodiment, various devices can be connected via the Internet (or another network) connection. That is, any one of connections 270 and 280 can represent several different computing devices and network components that allow various devices to communicate over the Internet through any of hard-wired connections or wireless connections. Any one of connections 270 and 280 can also be a combination of several connection modes.

[0065] Figure 2 The configuration of the devices in the system 200 is only one physical system on which the disclosed embodiments may be performed. Other configurations of the devices shown may exist to practice the disclosed embodiments. Further, other configurations of the devices shown may exist. Figure 2 The configuration of more or less devices than those shown in the figure can be used to practice the disclosed embodiments. In addition, Figure 2 The devices shown in the figure can be combined together to make the number of devices less than shown, or can be separated so that there are more than three devices in the system. It will be understood that various combinations of computing devices can perform the methods and systems disclosed herein. Examples of such computing devices may include other types of medical devices and sensors, infrared cameras / detectors, sensors that detect other parts of the electromagnetic spectrum, night vision cameras / detectors, other types of cameras, augmented reality goggles, virtual reality goggles, mixed reality goggles, radio frequency transmitters / receivers, smart phones, personal computers, servers, laptops, tablet computers, BlackBerry phones, RFID-enabled devices, smart watches or wearable devices, or any combination of these devices.

[0066] Return to reference Figure 1, the display 122 can be used to display various information about the patient 112 monitored by the system 100. In some embodiments, the system 100 (including the camera 114, the computing device 115, and the hardware processor 118) is used to generate sensor data (e.g., captured images and / or generated distance signals) and determine, via the signal processor, classifications of patient motion, which can be displayed in a user interface presented on the display 122, or otherwise indicated, such as with reference to Figure 2 Additionally or alternatively, the system 100 (including the camera 114, the computing device 115, and the hardware processor 118) is used to generate, receive, and / or display respiratory measurement data (e.g., respiratory rate) in the same or different user interface. Additionally or alternatively, the system 100 (including the camera 114, the computing device 115, and the hardware processor 118) is used to generate, receive, and / or display the generated sensor data as an image of the patient and / or ROI.

[0067] Figure 3 306 is a display view of an embodiment of a user interface 300 of a video-based patient monitoring system configured according to various embodiments of the present technology. In an embodiment, the user interface 300 may include one or more of the following: a visual representation of a patient 302, an overlay target rectangle 304, an ROI 306 of the patient 302 (e.g., the chest of the patient 302), a visual indicator 308 representing the respiratory region of the patient 302, a respiratory function measurement result 310 (shown as a respiratory volume signal), and an alignment indicator 312. In the illustrated embodiment, the user interface 300 includes a visual representation of the patient 302 and the ROI 306. In the illustrated user interface 300, the visual indicator 308 highlights the portion of the ROI 306 designated as the respiratory region. In an embodiment, the alignment indicator 312 may be used to indicate whether a sensor (e.g., a camera or other image capture device) is aligned in a predefined manner that does not satisfy a misalignment condition. In the illustrated embodiment, the alignment indicator 312 is only a simple indicator that can indicate whether the sensor is aligned relative to the patient environment by color, pattern, flashing frequency, or intensity (brightness) of light. Embodiments are contemplated in which the same or different indicators are additionally or alternatively presented on a surface of the sensor itself or a light indicator to aid in properly aligning the sensor relative to the patient environment (e.g., one or more of the ROI 306, the element represented by the visual indicator 308, and the patient 302). The visual indicator 308 can be an element of any device or stand-alone device that can allow a clinician to view the visual indicator 308 in a clinical environment and use the visual indicator to determine and / or correct for misalignment of the sensor relative to the patient environment.

[0068] Figure 4400A-400C are top views of embodiments of alignment 400A-400C of a patient 402 relative to a sensor. In a first alignment 400A, the sensor is aligned with the patient 402. The patient is located at the center of the superimposed target rectangle 404, and the axis bisecting the head and feet of the patient 402 substantially bisects the top and bottom boundaries of the superimposed target rectangle 404. This indicates that one or more of the distance between the sensor and the patient, the azimuth (rotation) angle between the sensor and the patient 402, the tilt angle between the sensor and the patient 402, and the translational position of the sensor relative to the patient 402 are aligned. Therefore, the pattern of the alignment indicator 406 indicates that the sensor is aligned relative to the patient 402.

[0069] In the second alignment 400B, the patient 402 is rotated relative to the superimposed target rectangle 404. This indicates a rotational misalignment, which can be corrected by a rotation angle (azimuth) correction. The system can detect (e.g., based on one or more of a predefined relationship, an aggregate alignment metric, and a sensed quality) that the rotational misalignment has occurred, and correspondingly indicate by an alignment indicator 406 that the sensor is misaligned relative to the patient.

[0070] In the third alignment 400C, the sensor is translated relative to the patient 402. This can be determined based on the patient 402 being outside (or otherwise not centered) of the overlay target rectangle 404. This translational misalignment causes the alignment indicator 406 to indicate that the sensor is not aligned with the patient 402.

[0071] A more complex alignment indicator 406 may indicate more than just whether the sensor is aligned relative to the patient 402. For example, the indicator may indicate one or more types of misalignment, one or more magnitudes of misalignment (e.g., one or more magnitudes of one or more types of misalignment), and corrective actions for correcting the misalignment (which may include type and / or magnitude data).

[0072] Figure 5 5 is a display view of an embodiment of an alignment indicator system 500 that may be presented on a sensor device, a display, a stand-alone device, or anywhere visible to a clinician in a clinical environment. The alignment indicator system 500 may include one or more of the following: rotational (azimuth) alignment indicators 502a, 502b, tilt angle alignment indicators 504a-504h, translational alignment indicators 506a-d, and distance alignment indicators 508a, 508b.

[0073] In the illustrated embodiment, alignment indicators 502, 504, 506, and 508 may be presented to show the type of misalignment, and additionally or alternatively, the type, direction, and / or magnitude of alignment corrections to be applied. For example, if certain elements are illuminated, they may indicate the type, direction, and / or magnitude of misalignment or corrections to correct the misalignment.

[0074] The rotation (azimuth) alignment indicators 502a, 502b each specify a different rotation direction and can indicate a rotational misalignment or a rotational misalignment correction. The tilt angle alignment indicators 504a-504h each specify a tilt angle misalignment or correction. For example, a sensor (e.g., a camera) can be attached to an articulated arm via a pivot. Movement of the sensor relative to the pivot (without rotation) will change the tilt angle of the sensor relative to the patient. The tilt angle alignment indicators 504a-504h can indicate the direction and magnitude of the misalignment or alignment correction. The translation alignment indicators 506a-d indicate a translational misalignment or a translational misalignment correction. For example, a translational misalignment correction can indicate how to manipulate (e.g., translate) an articulated arm coupled to the sensor to correct a translational misalignment. The distance alignment indicators 508a, 508b can indicate whether the distance between the sensor and the patient is incorrect (misaligned). In an embodiment where the distance alignment indicators 508a, 508b indicate misalignment correction, the first distance alignment indicator 508a illuminating may indicate that the distance between the sensor and the patient needs to be increased (by moving the articulated arm coupled to the sensor farther), while the second distance alignment indicator 508b illuminating may indicate the opposite.

[0075] One or more of the alignment indicators 502, 504, 506, and 508 may also present magnitude information to indicate the magnitude of the misalignment or misalignment correction. The magnitude information may include a flashing frequency of the light emitted by the indicator, an intensity or brightness of the light of the indicator, or a color of the light of the indicator (e.g., within a predefined representative scheme). In an embodiment, the alignment indicators 502, 504, 506, and / or 508 (whether on a sensor or on a display) may present a guide to allow a user to interpret the light emitted by the alignment indicators 502, 504, 506, and / or 508.

[0076] Embodiments are contemplated in which multiple indicators illuminate simultaneously to indicate that there is a misalignment or correction somewhere between the indicators. For example, translational alignment indicators 506a and 506b may illuminate simultaneously to indicate that there is a translational misalignment in a direction somewhere between the two. The magnitude information may be used to indicate in which direction the sensor is more misaligned among the directions indicated by translational alignment indicators 506a and 506b.

[0077] Embodiments are contemplated in which a single light indicator can indicate that any type of alignment is incorrect and can be presented in different ways depending on the direction and magnitude of the misalignment or correction. For example, the indicator can be a single rotating alignment indicator that presents a palette of colors that indicate one or more of the magnitude and direction of the misalignment. In another embodiment, the color can indicate the direction of the misalignment, and the flashing frequency of the light or the intensity of the light can indicate the magnitude of the indicated type of misalignment.

[0078] Figure 6 6 is a flow chart of an embodiment of a method 600 for qualifying the alignment of a sensor relative to a patient environment, configured in accordance with various embodiments of the present technology. In step 602, a respiratory region of the patient is determined. In an embodiment, a region of interest (ROI) may include a respiratory region of the patient. A respiratory region is a predefined region of a patient (e.g., chest, face, nostrils, etc.) whose movement is attributable to breathing. For example, a patient's chest typically moves during breathing. Movement of the chest cavity causes the lungs to expand. Respiratory movement is typically manifested as a continuous alternation of chest expansion and relaxation. Other ROIs may include the patient's hands, face, or other anatomical features. A respiratory region may be displayed on a display as a region indicator or "mask" on an image of the patient and / or the patient's environment. The respiratory region and region indicator may be determined as described in U.S. patent application Ser. No. 16 / 713,265, which is incorporated herein by reference.

[0079] In step 604, the sensor detects sensor data, including one or more of data representing multiple distances between the location of the sensor and the breathing zone and captured image data. In an embodiment, the sensor may be a depth sensing camera capable of detecting data representing multiple distances and / or captured image data.

[0080] In step 606, one or more misalignment qualities are generated based at least in part on the detected sensor data. In an embodiment, the one or more misalignment qualities may include one or more of an aggregate alignment metric and a sensing quality of the breathing zone.

[0081] In some embodiments, an alignment manager generates an aggregate alignment metric of alignment between image capture device(s) and a patient environment. In these embodiments, the alignment manager receives generated sensor data (e.g., captured images and / or sensed distances of elements of a patient environment, such as a patient's respiratory zone), and is operable to generate an aggregate alignment metric based at least in part on the detected sensor data. The aggregate alignment metric may include an aggregate distance or angle metric based on one or more of a mean, median, and truncated mean of detected distances between the image capture device(s) and the elements of the patient environment. The aggregate alignment metric may take into account multi-dimensional misalignment of different misalignment types, such as one or more of rotational misalignment, translational misalignment, tilt angle misalignment, and distance misalignment. The aggregate alignment metric may further take into account the magnitude and / or direction of the misalignment type.

[0082] An embodiment of the aggregated alignment metric may include an alignment score. In one embodiment, the alignment score is determined based on the aggregated distance between the (multiple) image capture devices and the determined respiratory region. For example, the alignment manager can determine the alignment score as in Equation 1. The aggregated distance may include one or more of the average, median, and truncated mean of the sensed distance data. In Equation 1, the absolute value is the aggregated distance minus one meter. If the distance differs too much from one meter, the alignment score will reflect that the (multiple) image capture devices are misaligned. In an embodiment, the alignment score will be further based on the aggregated angle, as presented in Equation 2. The aggregated angle may be based on one or more of the average, median, or mean of the angles detected between the (multiple) image capture devices and the determined respiratory region, or based on the difference in the distance vector of the position of the patient or sensor, which indicates the overall aggregated angle (e.g., different distances may indicate the overall angle of the sensor relative to the patient environment). The misalignment condition may be based on one or more of the alignment scores. In an embodiment, the misalignment condition may be met when the alignment score exceeds a threshold alignment score or falls outside an acceptable alignment score range.

[0083] In embodiments, the alignment manager additionally or alternatively determines a sensed quality of the respiratory region. In these embodiments, the alignment manager receives detected sensor data (e.g., a captured image of an element of a patient's environment (such as a patient's respiratory region) and / or a sensed distance to the element of the patient's environment) and is configured to determine the sensed quality of the determined respiratory region based on the detected sensor data through a predefined relationship between the sensor data and the sensed quality of the respiratory region. The sensed quality may include one or more of: a detected size or shape of the determined respiratory region, a detected fill ratio of an image representing the respiratory region (e.g., the extent to which there are gaps in a visual indicator overlaid on the image representing the respiratory region), a detected shape or size of a mask wearable by the patient. The alignment manager may then determine whether the sensed quality satisfies a misalignment condition. For example, the sensed shape or size of a sensed portion of the determined respiratory region may vary depending on the angle and / or distance at which the image capture device senses the respiratory region. If the sensed shape or size of the respiratory region is consistent with the predefined shape and / or size parameters of the misalignment condition, the sensed quality of the sensed shape or size may satisfy the misalignment condition.

[0084] In embodiments where the sensed size of a detected portion of the respiratory region is used to determine misalignment, the sensed size of the respiratory region relative to the size of a comparison region (e.g., one or more of the ROI, the entire respiratory region, an overlay target portion assigned to sensor data by a sensor or image processor when capturing sensor data) may indicate the distance between the image capture device(s) and the respiratory region. The ratio may be determined based on the number of pixels in the respiratory region relative to the number of pixels in the rest of the perceived environment (or within a predefined range of the field of view of the image capture device(s)), or the relative size may be determined by integrating depth or distance in the respiratory region or another indicative region (e.g., a region covering one or more of the patient's chest, head, arms, hands, other anatomical features that may demonstrate the orientation of the image capture device(s) relative to the patient) to generate a physical indicator of the size of the indicative region.

[0085] In embodiments where the sensed shape is a chest in the patient's breathing region, it may be contemplated that the sensed shape is substantially elliptical and substantially symmetrical about a bisector of the elliptical shape. Dissimilarity in size and / or shape may be used (e.g., as an element of a misalignment condition) to determine misalignment of the image capture device(s).

[0086] Additionally or alternatively, if a sensed fill ratio (ratio of a distance from a detected location in the determined breathing zone that does not meet the lighting condition and therefore cannot be presented as light in the breathing zone or mask to an absolute distance from a location in the determined breathing zone or other detected distance relative to the determined breathing zone that meets the lighting condition) is an element of a misalignment condition, the sensed quality of the fill ratio may meet the misalignment condition if the ratio falls within a predetermined range or meets a predetermined threshold for the misalignment condition. In these embodiments, sensing the fill ratio may be at least a portion of determining one or more alignment qualities.

[0087] In an embodiment, the predefined relationship between the generated sensor data and the sensing quality used to determine that the misalignment condition is satisfied can be represented by an inference model (such as a machine learning model). The inference model can be pre-trained (e.g., before making an inference determination) based on the generated labeled sensor data that is labeled to represent the sensing quality of the breathing area. For example, the inference model can be pre-trained by inputting time series data of images captured at different times (e.g., a first time and a second time) and / or generated distance signals, which are labeled with the labeled sensing quality of the determined breathing area. The inference model trainer can then compare the output of the inference model with the label associated with the input and determine the loss between the output and the associated label. The inference model trainer can then back-propagate or otherwise distribute the loss through the inference model (e.g., by adjusting one or more of the weights, activation functions, and biases represented in one or more nodes and edges of the neural network of the machine learning model). By repeating this process with different generated labeled sensor data, the inference model can represent the predefined relationship between the generated sensor data and the sensing quality.

[0088] The predefined relationships may be based on the patient's demographics. In an embodiment, the predefined relationships are established specifically for the patient's demographics by training the inference model exclusively or primarily on labeled data associated with the demographics. In another embodiment, the inference model is configured and / or pretrained to take demographic data as input, and the inference relationships within the inference model internally consider the demographic data.

[0089] In embodiments, the labeling and output of the inference model may include data (e.g., probabilities and / or statistics) representing a confidence in the output determination. In embodiments, predefined relationships including the inference model may be similarly applied between detected sensor data and one or more aggregated alignment metrics to determine aggregated alignment metrics based on the detected sensor data, which may be similarly pre-trained using sensor data labeled with the labeled aggregated alignment metrics. Further, embodiments are contemplated in which a single inference relationship may be determined based on detected sensor data, one or more sensed qualities, and one or more aggregated alignment metrics representing alignment between (multiple) image capture devices and a determined breathing zone.

[0090] Embodiments are contemplated in which the misalignment condition includes multiple considerations. For example, the alignment manager can determine one or more aggregate alignment metrics and one or more sensed qualities of the respiratory zone to determine whether the sensor data satisfies the misalignment condition indicating that the image capture device(s) are misaligned relative to the patient environment.

[0091] The system may additionally or alternatively determine other respiratory measurements, such as values ​​of various parameters (e.g., patient respiratory motion, patient non-respiratory motion, tidal volume, minute ventilation, respiratory rate, etc.). In an embodiment, the signal processor may determine the respiratory measurements based on the generated sensor data. In other embodiments, the signal processor may receive respiratory measurements generated by other means, including one or more of transthoracic impedance measurement, electrocardiogram, carbon dioxide analyzer, spirometer, pulse oximeter, and manual user input. Embodiments in which the signal processor does not process respiratory measurements other than patient motion classification are also contemplated. In an embodiment, the respiratory measurements may be considered in determining one or more of the aggregate alignment metric, determining the sensing quality, and the misalignment condition. For example, the misalignment condition may be based on the periodicity of the respiratory volume waveform. Irregular periodicity may indicate misalignment, while regular waveform periodicity may indicate correct alignment. Similarly, if the respiratory rate measurements and / or associated confidence values ​​of the respiratory rate measurements differ significantly from expected or predefined respiratory rate measurements (e.g., satisfy a dissimilarity condition), this may indicate that the image capture device(s) are misaligned relative to the patient environment.

[0092] In step 608, it is determined whether one or more misalignment qualities satisfy a misalignment condition. After determining one or more misalignment qualities (e.g., aggregate alignment metrics and / or sensing qualities), the alignment manager may determine whether one or more misalignment qualities satisfy one or more misalignment conditions based on the one or more misalignment qualities. In an embodiment, the misalignment condition may include a predefined threshold and / or a predefined value range representing the magnitude and / or direction of the misalignment type. In an embodiment, the aggregate alignment metric may include a score based on the value of the misalignment type. If the score is multi-factorial (e.g., based on values ​​of more than one alignment type or based on different methods of determining the nature and degree of alignment), the score may include a weighted average or geometric mean of each different factor.

[0093] In embodiments, the misalignment condition may be based at least in part on the relative orientation of a region proximal to the ROI. For example, the determined respiratory region may include a chest region and an ROI of one or more anatomical features, including but not limited to a determined ROI proximal to the head, arms, legs, or hands. If the head is not in the proper position directly above the torso, it may indicate that the image capture device(s) are rotationally misaligned (azimuthally misaligned).

[0094] In other embodiments, the misalignment condition may include hard cutoff values ​​for certain parameters. For example, even if the determination that one or more misalignment conditions are met is based on many factors, one or more of the factors may have a threshold or range where the misalignment condition will automatically fail. For example, the misalignment condition may include a maximum convergence distance of 1.5 meters between the image capture device(s) and the breathing zone. In this example, even if all other values ​​indicate that the image capture device(s) are aligned, the misalignment condition may be met based solely on the determination that the convergence distance is 1.6 meters, regardless of all other factors. In this embodiment, detecting that a determined breathing zone distance is greater than 1.5 meters indicates that a misalignment has occurred, regardless of other considerations.

[0095] In embodiments where the alignment manager determines one or more of an aggregate alignment metric, sensing quality, satisfaction of a misalignment condition (or a misalignment condition that the alignment manager uses to determine satisfaction), and a misalignment classification based on more than one of the above-described factors, these factors may be weighted and considered in a weighted sum, or these factors may be used to determine a geometric mean as an overall misalignment score or a score for a specific misalignment element (e.g., misalignment type, direction, and / or magnitude).

[0096] In embodiments, step 608 may further include a classification operation (not shown) of classifying one or more alignment qualities. In these embodiments, the alignment manager may include a misalignment classifier operable to classify misalignment characteristics based on satisfaction. The classified misalignment characteristics may include one or more of the following: misalignment of the (multiple) image capture devices relative to the determined breathing region, one or more types of misalignment (e.g., rotation, translation, tilt angle and / or distance), and one or more (e.g., one or more types of) misalignment magnitudes, etc. Embodiments are contemplated in which the classification is performed by an inferential relationship that is the same as or different from the inferential relationship that determines one or more of the aggregate alignment metric and / or sensing qualities of the determined breathing region. In embodiments using the same inferential relationship, the operation of determining the sensing quality and / or aggregate metric and the operation of classifying the misalignment characteristics may be combined into a single operation (e.g., the input is sensor data and the output is a misalignment classification).

[0097] In step 610, instructions for providing a misalignment notification are generated based at least in part on satisfying a misalignment condition and / or a classification of the misalignment (e.g., determined based on satisfying the misalignment condition). An alignment manager and / or an instruction generator for the alignment manager, executable by a hardware processor of the system, may generate instructions for displaying data generated by the alignment manager. The instructions may include data representing instructions to display the misalignment notification. The misalignment notification may include one or more of: an indication that the image capture device(s) are misaligned with the determined breathing region, a type of misalignment, a magnitude of the misalignment, and one or more corrective actions to be taken to align the image capture device(s) relative to the determined breathing region.

[0098] In embodiments, the instructions and / or indications may include data representing one or more of: a motion classification flag, a classification-specific display, an overlay display (e.g., configured to overlay a displayed element in a user interface to indicate misalignment relative to a determined respiratory zone), an image representation of patient motion (e.g., a captured image and / or a visual or video representation of a generated distance signal), an audio signal, a flashing element (e.g., wherein the amount of light of an element of a display alternately increases and decreases), different alarms, codes representing the above items, and the like. In embodiments where the classification is one of a plurality of classifications, each classification may correspond to a different display. The signal processor may output data representing the motion classification or a specific display associated with the alignment classification in a memory of the system. For example, different classifications may be represented in a display by different colors, different amounts of light in a display, different flashing frequencies in a display, and the like.

[0099] In embodiments where the misalignment classification represents different degrees of misalignment or confidence in its measurement, the corresponding display may represent a spectrum or range of colors, amounts of light, or flickering frequencies based on the magnitude of one or more of the different degrees of misalignment type and / or different degrees of confidence in determining the misalignment. In one embodiment, the instructions include activating one or more lights on the image capture device(s) (e.g., in the illustrated embodiment, a Figure 1 The light display may be simple and indicate the presence of misalignment by emitting a different color of light, activating a light only when the image capture device(s) are in one of alignment and misalignment, flashing a light only when the image capture device(s) are in one of alignment or misalignment, etc. The display may be more complex in that it indicates one or more of the type, direction, and magnitude of the misalignment and / or one or more of the type, direction, and magnitude of corrective action to correct the misalignment. See for examples of different indicator lights. Figure 5In embodiments, the magnitude of any particular type of misalignment may be represented by a frequency, color, and / or brightness of lights or pixels that represent a particular type or direction of misalignment or a particular type or direction of corrective action to correct the misalignment.

[0100] In an embodiment, the instructions for providing a misalignment notification may be configured to cause the display to display the misalignment notification as an element superimposed on or below (e.g., displayed behind) another displayed respiratory measurement result or image of the patient, including an ROI image for indicating the alignment of (multiple) image capture devices. For example, the display may be configured to display a measured respiratory rate, and may display an indication of a misalignment classification that exceeds or falls below the displayed respiratory rate. The superimposed or underlay display of the misalignment notification may be configured to form a visual contrast with the displayed respiratory measurement result. For example, these elements may have different colors and / or different transparencies, and / or the displayed misalignment notification may be at least partially transparent to maintain visibility of the displayed respiratory measurement result (e.g., as a highlight or patch of the displayed respiratory measurement result). Additionally or alternatively, the displayed misalignment notification may be superimposed on the ROI image. In an alternative embodiment, when the alignment manager determines that (multiple) image capture devices are misaligned, the instructions indicate that the display shall not display the respiratory measurement result and / or the patient image.

[0101] In embodiments, the misalignment manager may be communicatively coupled to a patient presence detector (e.g., hardware or software executable by one or more hardware processors) that detects the presence of a patient. In embodiments where the presence detector determines that a patient is not present, the misalignment manager may do one or more of the following: deactivate itself (e.g., until the presence of a patient is detected), deactivate any display or indication that the misalignment manager would instruct to display a misalignment indication, and transmit instructions to display a different alignment indication (e.g., color, light frequency, and / or intensity) that reflects the absence of a patient.

[0102] The above detailed description of the embodiments of the present technology is not intended to be exhaustive, or to limit the present technology to the exact form disclosed above. Although specific embodiments and examples of the present technology are described above for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present technology. For example, although the steps are presented in a given order, alternative embodiments may perform the steps in a different order. In addition, the various embodiments described herein may also be combined to provide additional embodiments.

[0103] The systems and methods described herein may be provided in the form of one or more tangible and non-transient machine-readable media (such as hard disk drives, hardware memories, etc.), on which instructions are recorded for execution by a processor or computer. An instruction set may include various commands that instruct a computer or processor to perform a specific operation (such as the methods and processes of the various embodiments described herein). An instruction set may take the form of a software program or application. Computer storage media may include volatile and non-volatile media and removable and non-removable media for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media may include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD or other optical storage devices, disk storage devices, or any other hardware media that can be used to store the required information and can be accessed by the components of the system. The components of the system may communicate with each other via wired or wireless communications. These components may be separate from each other, or various combinations of components may be integrated together into a monitor or processor, or contained in a workstation with standard computer hardware (e.g., processor, circuit, logic circuit, memory, etc.). The system may include processing devices such as microprocessors, microcontrollers, integrated circuits, control units, storage media, and other hardware.

[0104] According to the foregoing, it will be understood that the specific embodiments of the present technology have been described herein for illustrative purposes, but the well-known structures and functions have not been shown or described in detail to avoid making the description of the embodiments of the present technology unclear unnecessarily. In the event that any material incorporated herein by reference conflicts with the present disclosure, the present disclosure shall prevail. Where the context permits, singular terms or plural terms may also include plural terms or singular terms, respectively. In addition, unless the word "or" is explicitly limited to refer to a single item excluding other items in the case of a list with reference to two or more items, the use of "or" in such a list should be interpreted as including any single item in (a) the list, (b) all items in the list, or (c) any combination of items in the list. Where the context permits, singular terms or plural terms may also include plural terms or singular terms, respectively. In addition, the terms "comprising, including" and "having, with" are used throughout the text to mean at least including (multiple) stated features, so that any larger number of the same features and / or other features of additional types are not excluded. In addition, as used herein, the term "substantially" refers to the complete or almost complete scope or degree of an action, characteristic, property, state, structure, project or result. For example, an object that is "substantially" closed will mean that the object is completely closed or almost completely closed. In some cases, the exact degree of allowance for deviation from absolute completeness can depend on the specific context. However, in general, the proximity of completion will be to have the same overall result as having obtained absolute and complete completion. When used in a negative sense to refer to the complete or almost complete absence of an action, characteristic, property, state, structure, project or result, the use of "substantially" is equally applicable. As used herein, terms such as "substantially", "about", "approximately" or other relative degree terms are interpreted as those skilled in the art interpreting these terms and / or amounts as 1%, 2%, 3%, 4%, 5%, 6%, 7%, 8%, 9%, 10%, 11%, 12%, 13%, 14% or 15% of the measurement of the described quantitative or qualitative characteristics. The amplitude of one or more of the change. For example, the relative degree term "about the first time" indicates that time may have a certain amplitude of change relative to the first time. When values ​​and / or ranges of variation for particular features are set forth herein, ranges above those values, ranges below those values, and ranges between those values ​​are contemplated.

[0105] It should be understood that the various aspects disclosed herein may be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, depending on the examples, certain actions or events in any of the processes or methods described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., all described actions or events may not be necessary to perform the techniques). In addition, although for clarity, certain aspects of the present disclosure are described as being performed by a single module or unit, it should be understood that the techniques of the present disclosure may be performed by a combination of units or modules associated with, for example, a medical device.

[0106] In one or more examples, the described techniques can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored in the form of one or more instructions or codes on a computer-readable medium and can be executed by a hardware-based processing unit. A computer-readable medium may include a non-transitory computer-readable medium, which corresponds to a tangible medium, such as a data storage medium (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer).

[0107] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the above structures or any other physical structure suitable for implementing the described techniques. Furthermore, these techniques may be fully implemented in one or more circuits or logic elements.

Claims

1. A method for qualifying alignment of a sensor relative to a patient environment, the method comprising: Determine the patient's breathing zone; detecting sensor data by a sensor, the sensor data comprising a plurality of distances between a location of the sensor and the determined breathing zone; generating an aggregate alignment metric based at least in part on the detected sensor data; Determining that the aggregate alignment metric satisfies a misalignment condition; as well as Instructions to provide a misalignment notification are generated based at least in part on determining that the misalignment condition is satisfied.

2. The method of claim 1, wherein: The aggregated alignment metric includes an aggregated distance determined based on one or more of a mean, a median, and a trimmed mean of the plurality of distances.

3. The method of claim 1, wherein: The aggregated alignment metric includes a first alignment score based on one or more of: an aggregated distance and an aggregated angle determined based on the plurality of distances.

4. The method of claim 3, wherein: The aggregated alignment metric comprises an aggregated alignment score comprising a weighted average of at least the first alignment score and a second alignment score based on image data of the determined respiratory region captured by the sensor or a different sensor.

5. The method of claim 4, wherein: The second alignment score is based on a predefined relationship between the image data and an alignment of the sensor or the different sensor relative to the determined respiratory zone.

6. The method of claim 1, further comprising: A misalignment characteristic of the sensor relative to the determined breathing zone is classified based at least in part on satisfying the misalignment condition, wherein the generating the instructions to provide the misalignment notification are based at least in part on the classified misalignment characteristic.

7. The method of claim 6, wherein: The misalignment characteristic includes one or more of an indication of a size of the misalignment and an indication of a magnitude of the misalignment.

8. The method of claim 6, wherein: The instructions include a corrective action for modifying an alignment of the sensor relative to at least a portion of a respiratory zone of the patient based on the classified misalignment characteristic.

9. The method of claim 1, wherein: The instructions are configured to cause an externally visible indicator on the sensor to provide the misalignment notification.

10. The method of claim 1, wherein: The instructions are configured to cause an indicator on an external display to provide notification of the misalignment.

11. A method for qualifying alignment of a sensor relative to a patient environment, the method comprising: Determine the patient's breathing zone; detecting sensor data by a sensor, the sensor data comprising one or more of: a captured image of the breathing zone; and a plurality of distances between the location of the sensor and the breathing zone; determining a sensed quality of the determined breathing zone based on the detected sensor data by means of a predefined relationship between the sensor data and the sensed quality of the breathing zone; determining whether a sensing quality of the determined breathing region satisfies a misalignment condition; as well as Instructions to provide misalignment notification are generated based at least in part on satisfying the misalignment condition.

12. The method of claim 11, wherein: The respiratory region includes one or more of a region covering a portion of the patient's chest and a region covering at least a portion of the patient's face.

13. The method of claim 11, further comprising: A misalignment characteristic of the sensor relative to the determined breathing zone is classified based at least in part on satisfying the misalignment condition, wherein the generating the instructions to provide the misalignment notification are based at least in part on the classified misalignment characteristic.

14. The method of claim 11, wherein: The determined sensed quality of the breathing zone includes a detected breathing performance indicator.

15. The method of claim 11, wherein: The sensed quality of the determined breathing region includes one of a detected size and a detected shape of the determined breathing region.

16. The method of claim 11, wherein: The determined sensed quality of the breathing zone comprises a detected fill ratio of an image representing a mask wearable by the patient.

Images