Detection of patient occupancy using 3D passive stereo camera and point cloud processing using AI / ML algorithm

Abstract

The invention relates to patient occupancy detection. An apparatus is provided for processing data collected from an area within a facility for transmission to a cloud-based processing system for processing at the cloud-based processing system. The device comprises an input unit, a point cloud data generator and an output unit. The input unit is configured to receive data collected from an area in a health care institution, the data to be sent to a cloud-based processing system. The acquired data includes one or more of video data, image data, and ranging data. The point cloud data generator is configured to process the collected data to generate point cloud data that can be used to provide point cloud rendering of the region in the health care institution. The output unit is configured to communicate with a network to transmit the generated point cloud data and the corresponding metadata to the cloud-based processing system.

Description

Technical Field

[0001] The present invention relates to patient occupancy detection and, in particular, to an apparatus and method for processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system, a cloud-based processing system, a system, a computer program product, and a computer-readable medium. Background Art

[0002] In a multi-vendor and multi-modal environment, it would be a challenge to identify scanner room occupancy without being connected to an HL7 feed or integrated with multi-vendor radiology information system (RIS) or healthcare information system (HIS) applications. Scanner room occupancy and scan progress updates would help identify the remaining time in an examination and improve the scheduling of scanners, personnel, and resources, and assist other workflow optimization solutions in applications such as a cloud-based platform radiology operations command center (ROCC).

[0003] Currently, patient occupancy in a scanner room is determined by processing the video stream of a camera installed inside or outside the scanner room using an edge device such as a tablet, which requires powerful image or video processing capabilities with strong computing power, significantly increasing the hardware cost. Processing images in the cloud is a viable alternative to local processing; however, transmitting images of patients or staff to the cloud may violate the General Data Protection Regulation (GDPR) privacy regulations. Summary of the Invention

[0004] Therefore, there may be a need to provide a method for improved room / patient occupancy detection.

[0005] The object of the present invention is solved by the subject matter of the independent claims, wherein further embodiments are incorporated in the dependent claims. It should be noted that the aspects described below of the present invention also apply to an apparatus and method for processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system, a cloud-based processing system, a system, a computer program product, and a computer-readable medium.

[0006] According to a first aspect of the present disclosure, there is provided an apparatus for processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system. The apparatus includes an input unit, a point cloud data generator, and an output unit. The input unit is configured to receive data collected from an area in a healthcare facility, the data to be sent to a cloud-based processing system. The collected data includes one or more of video data, image data, and ranging data. The point cloud data generator is configured to process the collected data to generate point cloud data, which can be used to provide a point cloud rendering of the area in the healthcare facility. The output unit is configured to communicate with a network to transmit the generated point cloud data and corresponding metadata to the cloud-based processing system. The generated point cloud data is preferably capable of being used to determine the presence, location, and / or activity of: (i) one or more persons present in an area of a healthcare facility and / or (ii) one or more objects present in an area of a healthcare facility.

[0007] Accordingly, a solution architecture is proposed that utilizes sensor devices to provide sensor data such as video data, image data, and / or ranging data. The sensor data is converted by a point cloud data generator into point cloud data. The simplified data is transmitted to a remote location, namely a cloud-based processing system, where the presence, location, and / or activity of persons and / or objects can be determined based on the rendering.

[0008] The methods described herein are designed to protect the privacy of patients and employees and can be used in applications such as ROCC. Additionally, the methods described herein require little integration with HIS / RIS solutions available in hospitals and can be applied to any type of imaging modality. Further, since only point cloud data is captured rather than images or videos of an entire room, small processing edge devices are sufficient to capture the point cloud data and send it to the cloud for predictive insights. The present disclosure proposes a method for, for example, room / patient occupancy detection, where the processing performance requirements for edge devices are minimal and the main processing is assigned to the cloud. The apparatus described herein can be implemented as an existing edge device in a healthcare facility or implemented within an existing edge device in a healthcare facility, thus eliminating the need for other devices in a hospital facility such as tablets, personal computers (PCs), and servers.

[0009] According to an embodiment of the present disclosure, an area in a healthcare facility includes one or more of the following: an imaging room, a treatment room, a patient preparation room, a patient waiting room, and a patient transfer room.

[0010] According to an embodiment of the present disclosure, the apparatus further includes a point cloud data extractor. The point cloud data extractor is configured to: extract a subset of data from the generated point cloud data, the subset of data being usable to determine the presence, location, and / or activity of (i) one or more persons in an area within a healthcare facility, and / or (ii) one or more objects in an area within a healthcare facility; and provide the extracted point cloud data to a cloud-based processing system.

[0011] Thus, the apparatus according to this embodiment can extract and transmit a relevant subset of data to the cloud, where appropriate cloud services (e.g., machine learning services) are applied in the cloud to obtain inferences. For example, only the point cloud information of skeletal information can be extracted and sent to the cloud for further processing, thus providing more efficient data transmission.

[0012] This will be described in detail below, particularly with regard to Figure 3 the point cloud data extractor shown in

[0013] According to an embodiment of the present disclosure, the subset of the extracted data includes one or more of the following: point cloud data of one or more persons present in an imaging modality room, point cloud data of one or more objects present in an imaging modality room, one or more data points having coordinate changes, and one or more data points that are members of a defined point class.

[0014] This will be described in detail below, particularly with regard to Figure 3 the point cloud data extractor shown in

[0015] According to an embodiment of the present disclosure, the one or more objects include one or more of a scanner table, a wearable device, a mobile / movable device, and an attachable device.

[0016] According to an embodiment of the present disclosure, the point cloud data extractor is configured to adjust one or more of the resolution, size, and sampling rate of the generated point cloud data based on a stage of a clinical workflow, and provide the adjusted point cloud data to a cloud-based processing system.

[0017] Thus, the apparatus according to this embodiment can allow for dynamic adjustment of the data transmitted and processed to account for different stages of a clinical workflow.

[0018] This will be described in detail below, particularly with regard to Figure 3 the point cloud data extractor shown in

[0019] According to an embodiment of the present disclosure, the point cloud data extractor is configured to: adjust one or more of the resolution, size, and sampling rate of the generated point cloud data based on the processing requirements of the cloud-based processing system, and provide the adjusted point cloud data to the cloud-based processing system.

[0020] Thus, the apparatus according to this embodiment can allow for dynamically adjusting the data being transmitted and processed to account for different usage scenarios (e.g., quick detection versus detailed status detection).

[0021] This will be described in detail below, particularly with regard to Figure 3 the point cloud data extractor shown in

[0022] According to a second aspect of the present disclosure, a cloud-based processing system is provided. The cloud-based processing system includes at least one communication interface and an analysis engine. The at least one communication interface is configured to communicate with a network to receive point cloud data providing a point cloud rendering of an area in a healthcare facility. The analysis engine is configured to apply at least one data analysis algorithm to determine the presence, location, and / or activity of the following based on the received point cloud data: (i) one or more persons present in the healthcare facility area, and / or / (ii) one or more objects present in the area of the healthcare facility.

[0023] This will be explained in detail below, particularly for Figure 4 the example shown in

[0024] According to an embodiment of the present disclosure, at least one data analysis algorithm includes a self-learning algorithm configured to continuously learn based on one or more of the received point cloud data, workflow data, device log data, workflow-related data, and user feedback.

[0025] According to an embodiment of the present disclosure, the analysis engine is configured to apply (i) at least one data analysis algorithm or (ii) a rule-based model and / or a symmetry-based model to identify and fill in missing data.

[0026] According to an embodiment of the present disclosure, the cloud-based processing system further includes a data storage device configured to store the received point cloud data and metadata.

[0027] According to a third aspect of the present disclosure, a system is provided. The system includes a sensor device including one or more sensors configured to collect one or more of video data, image data, and ranging data from an area in a healthcare facility, an apparatus according to the first aspect of the present disclosure and any related examples, and a cloud-based processing system according to the second aspect of the present disclosure and any related examples.

[0028] This will be explained in detail below, especially with respect to Figure 1 、 Figure 2 and Figure 5 the example systems shown in

[0029] According to a fourth aspect of the present invention, there is provided a method of processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system, the method comprising:

[0030] Receiving data collected from an area in a healthcare facility to be sent to a cloud-based processing system, wherein the collected data includes one or more of video data, image data, and ranging data;

[0031] Processing the collected data to generate point cloud data, the point cloud data being usable to provide a point cloud rendering of the healthcare facility area; and

[0032] Communicating with a network to transmit the generated point cloud data and corresponding metadata to a cloud-based processing system, wherein the generated point cloud data is preferably usable to determine the presence, location, and / or activity of: (i) one or more persons present in the area of the healthcare facility and / or (ii) one or more objects present in the area of the healthcare facility.

[0033] This will be explained in detail below, especially with respect to Figure 6 the example method shown in

[0034] According to another aspect of the present disclosure, there is provided a computer program product comprising instructions which, when run by a processing unit, cause the processing unit to perform the steps of the method according to the third aspect of the present disclosure and any associated examples.

[0035] According to another aspect of the present disclosure, there is provided a computer-readable medium having stored thereon the computer program product.

[0036] It should be understood that all combinations of the above concepts and additional concepts discussed in more detail below (assuming these concepts are not mutually inconsistent) are expected to be part of the inventive subject matter disclosed herein. In particular, all combinations of the subject matter of the claims are expected to be part of the inventive subject matter disclosed herein.

[0037] With reference to the embodiment(s) described below, these and other aspects of the invention will become apparent and be elucidated. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In the drawings, like reference numerals generally refer to like parts in different views. Also, the drawings are not necessarily to scale, but rather are focused on illustrating the principles of the present invention.

[0039] Figure 1 Schematically illustrates an exemplary environment that may be suitable for implementing the method of the present invention.

[0040] Figure 2 Schematically illustrates another exemplary environment that may be suitable for implementing the method of the present invention.

[0041] Figure 3 Schematically illustrates an example of components that may be present in the device.

[0042] Figure 4 Schematically illustrates an example of components that may be present in a cloud-based processing system.

[0043] Figure 5 Illustrates an example of a multi-vendor and multi-mode environment that may be suitable for implementing the present method.

[0044] Figure 6 Illustrates a flowchart of a method for describing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system. Detailed Description

[0045] Figure 1 Schematically shows an exemplary environment 100 that may be suitable for implementing the method of the present invention. Exemplary environment 100 includes a sensor device 120, a device 130, a network 140, and a cloud-based processing system 150. In the illustrated example, both the sensor device 120 and the device 130 are located within a healthcare facility 110. The healthcare facility 110 can be any location that provides healthcare. Examples of healthcare facilities particularly include small clinics, physician offices, urgent care centers, and large hospitals with well-equipped emergency rooms and trauma centers.

[0046] The sensor device 110 of the illustrated example may include one or more sensors, which may be one or more devices, modules, or subsystems, aimed at detecting environmental activities, events, and / or changes at one or more locations of the healthcare facility 110 and sending the collected sensor data to some other devices, modules, subsystems, etc. The collected sensor data may include one or more of video data, image data, and ranging data. Examples of such sensors particularly include image capture devices (e.g., cameras), light detection and ranging (LiDAR) sensors, depth sensors, optical light sensors, etc. The sensor data is not limited to the visible spectral range. Examples of sensor data particularly include grayscale images, near-infrared (NIR) images, RGB images, multispectral images, hyperspectral images, etc. The ranging data may be collected using, for example, a stereo or multi-camera setup system, standard time-of-flight (TOF) technology, or any other suitable technology.

[0047] The device 130 in the illustrated example generally may include various physical and / or logical components for transmitting and manipulating information, which may be implemented as hardware components (e.g., computing devices, processors, logic devices), executable computer program instructions executed by various hardware components (e.g., firmware, software), or any combination thereof, as required by a given set of design parameters or performance constraints. In some implementations, the device 130 may include one or more microprocessors or computer processors that run appropriate software. The software may have been downloaded and / or stored in a corresponding memory, e.g., a volatile memory such as RAM or a non-volatile memory such as flash memory. The software may include instructions for configuring one or more processors to perform the functions described herein. It should be noted that the device 130 may be implemented with or without a processor and may also be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) for performing other functions. For example, at least one processing unit may be implemented in the device or apparatus in the form of programmable logic (e.g., a field-programmable gate array (FPGA)).

[0048] In some examples, as Figure 1 shown, the device 130 may be implemented as an edge device or embodied in an edge device. The term "edge device" refers, at least in some examples, to a computing node that performs edge computing operations. In some examples, an edge device may be referred to as an "edge node" or "edge system," regardless of whether it operates as a client, server, or intermediate entity. In some examples, the device 130 may be incorporated into a server, gateway, local (on premise) unit, or end-consumer device, etc.

[0049] In some other examples, asFigure 2 As shown in, device 130 may be implemented as a device coupled to edge device 160, which has one or more communication interfaces configured to send data to and receive data from cloud-based processing system 150 via network 140.

[0050] Although device 130 is shown in Figure 1 and Figure 2 as a hardware system, it should be understood that in alternative embodiments, device 130 may be implemented as a software system (e.g., software residing in the edge device) that instructs the hardware to perform operations.

[0051] Reference Figure 3 illustrates device 130 in more detail.

[0052] Network 140 may represent a network such as the Internet, a wireless local area network (WLAN), or a wireless wide area network (WWAN) (including proprietary and / or enterprise networks for companies or organizations), a cellular core network (e.g., an evolved packet core (EPC) network, a next-generation packet core (NPC) network, a 5G core (5GC), or some other type of core network), a cloud computing architecture / platform providing one or more cloud computing services, and / or a combination thereof. As an example, network 140 and / or the access technology may include cellular technologies such as LTE.

[0053] Cloud-based processing system 150 may represent one or more application servers, a cloud computing architecture / platform providing computing services, and / or some other remote infrastructure. Cloud computing services (also known as cloud services) are one or more functions provided via cloud computing that are invoked using defined interfaces (e.g., APIs, etc.). Cloud-based processing system 150 may include any of a number of services and functions, such as occupancy detection (e.g., overall availability of a room or instantaneous presence and number of people, etc.) and more detailed analytics (e.g., number and interactions of people, patient size and profile, etc.).

[0054] Cloud-based processing system 150 will be described in more detail with respect to the Figure 4 example in.

[0055] Figure 3Illustrated is an example of components that may be present in apparatus 130 for implementing the techniques described herein. Apparatus 130 may include any combination of the hardware or logic components cited herein, and it may include or be coupled to any device that may be used with a communication network or a combination of such networks. These components may be implemented as ICs adapted in apparatus 130, portions thereof, discrete electronic devices or other modules, instruction sets, programmable logic or algorithms, hardware, hardware accelerators, software, firmware, or combinations thereof, or as components incorporated within the chassis of a larger system.

[0056] The illustrated example of apparatus 130 includes an input unit 132, a point cloud data generator 134, a point cloud data extractor 136, and an output unit 138. In alternative configurations, different and / or additional components may be included in apparatus 130. Additionally, in some examples, the functions described in connection with Figure 3 one or more of the components shown in Figure 3 may be distributed among the components in a manner different from that described in connection with

[0057] The input unit 132 and the output unit 138 may be implemented as circuits and / or components to facilitate input / output operations with the point cloud data generator 134 and the point cloud data extractor 136. For example, the input unit 132 and the output unit 138 may be implemented as, or include, a memory controller hub, an input / output control hub, an integrated sensor hub, a firmware device, a communication link (e.g., a point-to-point link, a bus link, a wire, a cable, an optical waveguide, a printed circuit board trace, etc.), and / or other components and subsystems to facilitate input / output operations. In some examples, the input unit 132 and the output unit 138 may form part of an SoC and be incorporated within a computing subsystem.

[0058] The point cloud data generator 134 and the point cloud data extractor 136 may be implemented as integrated circuits, embedded systems, FPGAs, systems on a chip (SoCs), or other integrated systems or devices. In some implementations, the point cloud data generator 134 and the point cloud data extractor 136 include or embody a processor and a memory. The processor may be implemented as any type of processor capable of performing the functions described herein (e.g., executing an application). For example, the processor may be implemented as (one or more) multi-core processing, a microcontroller, or other processor or processing / control circuitry. The processor may be implemented as, include, or be coupled to an FPGA, an application specific integrated circuit (ASIC), reconfigurable hardware, or a hardware circuit, or other dedicated hardware to facilitate the execution of the functions described herein.

[0059] In some examples, output unit 138 may include or be coupled to a communication circuit, which may be implemented as any communication circuit, device, or collection thereof capable of enabling communication between device 130 and cloud-based processing system 150 over a network. The communication circuit may be configured to use any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., cellular network protocols (such as 3GPP 4G or 5G standards), wireless local area network protocols (such as IEEE802.11 / ), wireless wide area network protocols, Ethernet, Bluetooth Low Energy, Internet of Things protocols (such as IEEE802.15.4 or ), low power wide area network (LPWAN) or low power wide area (LPWA) protocols, etc.) to effectuate such communication.

[0060] In use, device 130 receives, via input unit 132, sensor data collected by sensor device 110, which may be used to detect activities, events, and / or environmental changes at one or more locations of healthcare facility 110. The sensor data may include video data, image data, and / or range data.

[0061] Point cloud data generator 134 processes the collected sensor data to generate point cloud data and corresponding metadata, which can be used to provide a point cloud rendering of an area in a healthcare facility. The point cloud data includes a collection of points referred to as a point cloud. The point cloud includes a collection of individual three-dimensional points. Each point may contain many other attributes in addition to having a three-dimensional (x, y, z) position, such as color, reflectivity, surface normal, etc. The metadata may include additional information about the point cloud data, such as the reliability of the point coordinates, tracking vectors depending on the underlying point cloud generation technology, etc.

[0062] For example, the sensor device 110 can include multiple video or imaging cameras that are arranged to capture color attributes in scenes at one or more locations in a healthcare facility. Various methods can be used to capture the position of an object and / or a person, including in particular infrared depth cameras, photogrammetry and stereo parallax, and the use of structured light or laser illumination of the scene. The acquired sensor data is then processed using algorithms to obtain a sparse vowelized point cloud representative of the captured scene. An example of a method for capturing a voxelized point cloud using only cameras is described in C. Loop, C. Zhang, and Z. Zhang, "Real-time high-resolution sparse voxelization with application to image-based modeling", Proc. 5th High-Perform. Graph. Conf., pp. 73-79, 2013.

[0063] As an option, the point cloud data generated by the point cloud data generator 134 can be provided as an input to the point cloud data extractor 136, which is configured to extract a subset of data from the generated point cloud data that can be used to determine the presence, location, and / or activity of one or more persons present in an area in a healthcare facility, and provide the extracted point cloud data to a cloud-based processing system. Alternatively or additionally, the point cloud data extractor 136 can be configured to extract a subset of data from the generated point cloud data that can be used to determine the presence, location, and / or activity of one or more objects present in an area in a healthcare facility, and provide the extracted point cloud data to a cloud-based processing system.

[0064] The extracted subset of data may include one or more of the following: (i) point cloud data of one or more persons present in an imaging modality room, (ii) point cloud data of one or more objects present in an imaging modality room, (iii) one or more data points whose coordinates have changed, and (iv) one or more data points that are members of a defined point class.

[0065] For example, the point cloud data extractor 136 can extract a subset of points from the generated point cloud based on the indices output by the segmentation algorithm, where the indices associate each pixel or voxel in the acquired video, image, and / or range data with a class label, such as a person (e.g., a patient) or an object (e.g., a scanner table, a wearable device, a mobile / movable, attachable device, etc.). Examples of such segmentation algorithms include U-Net, DeepLab, Convolutional Neural Network (CNN), etc. For example, the extracted subset of points may only contain skeleton information. The skeleton of a shape, especially an articulated shape such as a person or an object, provides an intuitive and effective abstraction that helps determine the presence, location, and / or activity of one or more people and / or objects at one or more locations in a healthcare facility.

[0066] In some examples, the sensor device can include one or more cameras that support a Software Development Kit (SDK). Using the SDK supported by the cameras, the point cloud data extractor 134 can extract only the point cloud information for the skeleton information from the generated point cloud.

[0067] In some examples, the point cloud data extractor 136 can extract a subset of points representing the boundary of the scanner table from the point cloud data generated by the point cloud data generator 134 so that the table movement can be inferred. For example, the point cloud data extractor 136 can extract a subset of points representing, for example, a wearable device, a mobile / movable device, and an attachable device (e.g., a coil) from the point cloud data generated by the point cloud data generator 134 so that the patient movement can be inferred.

[0068] Such a simple criterion can be to transmit only those data points whose coordinates have changed. In other words, in some examples, the point cloud data extractor 136 can extract a subset of points representing one or more data points with coordinate changes from the point cloud data generated by the point cloud data generator 134, such that the movement of people and / or objects can be inferred.

[0069] As an alternative, the extraction of the subset can be dynamically adjusted, such as the resolution, size, and / or sampling rate of the point cloud data stream.

[0070] In one example, according to the specific use case of the occupancy data (e.g., the general availability of a room or the instantaneous presence and number of people), the extraction of the subset, such as the resolution, size, and / or sampling rate, can be adjusted. Specifically, a low-resolution point cloud data stream can be used for fast, low-latency analysis, e.g., to obtain information about whether someone is present. A high-resolution and higher-latency point cloud data stream can be used to obtain slower but more detailed analysis results, such as the number of people and interactions, patient size and profile, etc.

[0071] In another example, the use and activation of these streams can be dynamically adjusted based on the stage of the clinical workflow and / or other available information. For example, when no person is present or moving in the scene detected by the sensor device, a low-resolution point cloud data stream can be transmitted to the cloud-based processing system. On the other hand, when a patient contour needs to be generated to guide the patient's positioning on the operating table, a high-resolution and higher-latency point cloud data stream can be transmitted to the cloud-based processing system for slower but more detailed analysis.

[0072] Figure 4 Illustrated is an example of components that may exist in a cloud-based processing system 150 for implementing the techniques described herein. The cloud-based processing system 150 may include any combination of the hardware or logic components cited herein and may include or be coupled to any device capable of being used with a communication network or a combination of such networks.

[0073] The illustrated example of the cloud-based processing system 150 includes a communication interface 152, an analysis engine 154, and a data storage device 156.

[0074] The communication interface 152 may be a hardware element or a collection of hardware elements for communicating on one or more networks (e.g., network 140) and / or communicating with other devices (e.g., device 130 or edge device 160). The communication interface 152 may be configured to use any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., cellular network protocols (such as 3GPP 4G or 5G standards), wireless local area network protocols (such as IEEE 802.11 / )), wireless wide area network protocols, Ethernet, low-power Bluetooth, Internet of Things protocols, such as IEEE 802.15.4 or low-power wide area network (LPWAN) or low-power wide area (LPWA) protocols, etc.) to implement such communication.

[0075] The analysis engine 154 provides one or more cloud computing services to analyze the point cloud data received from the device 130 and transmit the analysis results to the device 130 via the communication interface 152. Examples of such services include, among others, patient occupancy analysis, room availability analysis, activity detection, etc. The cloud computing services provided by the analysis engine 154 may include machine learning (ML) services and / or other similar services for performing various analyses. The term "machine learning" or "ML" refers, at least in some examples, to a computer system using implemented algorithms and / or statistical models to perform one or more specific tasks without using explicit instructions, but relying on patterns and inferences. ML algorithms build or estimate one or more mathematical models (referred to as "ML models", etc.) based on sample data (referred to as "training data", "model training information", etc.) in order to make predictions or decisions without being explicitly programmed to perform such tasks. Generally speaking, an ML algorithm is a computer program that learns from experience regarding a certain task and a certain performance metric, and an ML model can be any object or data structure created after training an ML algorithm using one or more training data sets. After being trained, the ML model can be used to make predictions on new data sets. Although the term "ML algorithm" refers to a different concept than the term "ML model" at least in some embodiments, these terms discussed herein can be used interchangeably for the purposes of this disclosure.

[0076] In some examples, a self-learning ML model can be deployed to one or more MR services. For example, the data storage device 156 can store the point cloud data received from the device 130 in a blob-type memory. From the data storage 156, the point cloud data is input into the self-learning model, which is regularly deployed to the ML service. This solution may support self-learning AI model components that continuously learn based on input point cloud data from different modalities and different hospital facilities, and that continuously learn to strengthen their models and provide more reliable and robust results. Figure 5 An example of the deployment of a self-learning ML model is illustrated.

[0077] In addition, these trained ML models can identify and automatically fill in missing data caused by occluded and truncated objects when extracting point cloud data. Alternatively, these ML models can be replaced by rule-based and symmetry-based models to automatically fill in the missing data.

[0078] Figure 5 An example of a multi-vendor and multi-modal environment 300 suitable for implementing the present method is shown. The exemplary environment 300 includes a sensor device 120, a device 130, and a cloud-based processing system 150. The exemplary environment 300 includes a sensor device 120, a device 130, a network 140, and a cloud-based processing system 150.

[0079] In the illustrated example, the environment 300 includes one or more scanner rooms. In each scanner room, there is one or more medical imaging modalities for generating medical imaging data for a patient. Although Figure 5 three medical imaging modalities are illustrated by way of example, it should be understood that fewer or more medical imaging modalities than the examples described herein may be used for a particular implementation. Examples of medical imaging modalities may include, but are not limited to, magnetic resonance imaging (MRI), ultrasound, medical radiation, angiography, and computed tomography (CT) scanners. Medical imaging modalities can be used in fields including, but not limited to, radiology, cardiology, oncology, nuclear medicine, radiotherapy, neurology, orthopedics, obstetrics, gynecology, ophthalmology, dentistry, maxillofacial surgery, dermatology, pathology, clinical trials, veterinary medicine, and medical / clinical photography, etc.

[0080] The sensor device 120 is used to record activities, events, and / or changes within one or more scanner rooms and provide sensor data including information about the detected activities, events, and / or changes to the device 130. The sensor device may include one or more sensors, such as Figure 5 the illustrated sensors 122, 124, and 126, which may be mounted inside or outside the scanner room. Although Figure 5 three sensors 122, 124, and 126 are shown by way of example, it should be understood that fewer or more sensors than the examples described herein may be used for a particular implementation. The sensor data includes video data, image data, and / or range data. In some examples, the sensor device may include one or more sensors that have built-in algorithm capabilities to detect objects and also support depth sensing.

[0081] The illustrated example of the device 130 is embodied as or embodied in an edge device that sends data to or receives data from the cloud-based processing system 150. The device 130 includes a sensor data interface 132, a point cloud data generator 134, a point cloud data extractor 136, and a communication interface 136.

[0082] The sensor data interface 132 may include one or more physical ports, such as USB, Bluetooth, Ethernet, wireless Ethernet, etc., for communicating with the sensor device 110 to receive the collected sensor data.

[0083] The point cloud data generator 134 processes the acquired data to generate point cloud data, which can be used to provide a point cloud rendering of an area in a healthcare facility. Since only the point cloud data is captured instead of a full room image and / or video, a small processing edge device is sufficient to capture the point cloud data and send it to the cloud-based processing system 150, e.g., to build an ML self-learning model to predict insights.

[0084] In some examples, only a subset of the point cloud data is sent to the cloud-based processing system 150, which is sufficient to, for example, build an ML self-learning model to predict insights. In such examples, the point cloud data extractor 136 can extract a relevant subset from the point cloud data generated by the point cloud data generator 134. For example, features extracted from the point cloud can be used to identify patients, staff, and other people, and the data can be sent to the cloud-based processing system 150 for further processing. As a further example, using a camera-supported SDK, only the point cloud information for skeletal information can be extracted and sent to the cloud-based processing system 150 for further processing. The idea of only extracting the point cloud relative to the skeletal image is to avoid transmitting less relevant data to the cloud, e.g., not sending the point cloud data of the entire scene that requires a large amount of processing and data transmission to the cloud-based processing system. Instead, some key points are sent, such as the boundaries of the scanner table, so that the movement of the table can be inferred. This also applies to coils. In addition to this, there are other methods aimed at reducing data transmission to the cloud. For example, a simple criterion can be to only transmit those data points whose coordinates have changed. This method may improve the efficiency and latency of the solution.

[0085] In some examples, the extraction of the subset can be adjusted according to a specific use case of the occupancy data, such as resolution, size, and / or sampling rate. For example, a low-resolution point cloud data stream can be used for fast and low-latency analysis, e.g., to acquire information on whether someone is present, while a high-resolution point cloud data stream can be used for slower and higher-latency analysis to obtain more detailed analysis results (e.g., the number of people and people's interactions, patient size and profile). The use and activation of these streams can be dynamically adjusted according to the stage of the clinical workflow and / or other available information. For example, a high-resolution point cloud data stream can be used to generate a patient profile to guide patient positioning on the operating table.

[0086] Then, the point cloud data is sent to the cloud-based processing system 150 via the communication interface 138 for further processing.

[0087] The cloud processing system 150 includes a communication interface 152, an analysis engine 154, and a data storage device 156. The communication interface 152 is configured to communicate with a network to receive point cloud data from the device 130. The data storage device 156 can store the point cloud data in a blob type of memory. From the data storage device 156, the point cloud data can be input into a self-learning model provided by the analysis engine 154, and the model is regularly deployed to the ML service. These self-supervised models will be regularly evaluated and deployed in one or more ML services to achieve performance improvement. These ML services may include various cloud services using existing models, such as room availability detection, patient occupancy detection, etc. In addition, these trained ML models can identify and automatically fill in missing data caused by occluded and truncated objects when extracting point cloud data. Alternatively, these ML models can be replaced by rule-based and symmetry-based models to automatically fill in the missing data. Activity detection based on point cloud data is a potential use case for self-supervised learning. Identifying the actions involved by the detected individuals can help clarify their roles (e.g., patients and staff) and help identify the workflow stages ongoing in the scanner room.

[0088] The predicted insight information will be used by the ROCC application for users to view and plan further arrangements for rooms, personnel, and other resources, etc.

[0089] Figure 6 A flowchart illustrating a method 400 for processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system is shown. The method can be used in combination with other methods and systems described herein. Specifically, Figure 6 The method 400 shown can be implemented in Figure 1 、 Figure 2 and Figure 5 the exemplary environments shown.

[0090] The method can be at least partially implemented by a computer and can be implemented in software or hardware, or software and hardware. In addition, the method can be executed by computer program instructions running on a unit providing data processing functions. The data processing device can be a suitable computing unit, such as an electronic control module, etc., or a distributed computer system. The data processing device or computer can respectively include one or more processors, memories, data interfaces, etc.

[0091] At block 410, the method includes the step of receiving the collected data to be sent to the cloud-based processing system from an area in a healthcare facility. The collected data includes one or more of video data, image data, and ranging data. When method 200 is in Figure 3When implemented on the apparatus shown, the data interface 132 can be used to implement block 410, and the data interface 132 is communicatively coupled to the sensor device to receive the collected sensor data.

[0092] At block 420, the method includes the step of processing the collected data to generate point cloud data, and the point cloud data can be used to provide a point cloud rendering of an area in a healthcare facility. This can be achieved by Figure 3 the point cloud data generator 134 shown.

[0093] Optionally, only a subset of the point cloud data is sent to the cloud-based processing system 150 for further processing to avoid transmitting less relevant data to the cloud, because sending the point cloud data of the entire scene and video may require a large amount of processing and data transmission to the cloud-based processing system 150. When method 200 is implemented on Figure 3 the apparatus shown, the point cloud data extractor 136 can be used to implement this optional step.

[0094] At block 430, the method includes the step of communicating with a network to transmit the generated point cloud data and corresponding metadata to a cloud-based processing system for further processing. In some examples, the generated point cloud data is preferably used to determine the presence, location, and / or activity of one or more persons present in the healthcare facility area. Alternatively or additionally, the generated point cloud data is preferably used to determine one or more objects present in the healthcare facility area. This can be achieved by Figure 3 the output unit 138 shown, which can include or be coupled to a communication interface to send data to and receive data from the cloud-based processing system 150.

[0095] In another exemplary embodiment of the present invention, a computer program or computer program unit is provided, characterized in that it is adapted to execute the method steps of the method according to one of the foregoing embodiments on a suitable system.

[0096] The computer program unit can thus be stored on a computing unit, which can also be part of an embodiment of the present invention. The computing unit can be adapted to execute the steps of the above method or cause the execution of the steps of the above method. In addition, it can be adapted to operate the components of the above apparatus. The computing unit can be adapted to automatically operate and / or execute the commands of the user. The computer program can be loaded into the working memory of the data processor. The data processor can thus be equipped to implement the method of the present invention.

[0097] This exemplary embodiment of the present invention superimposes a computer program that initially uses the computer program of the present invention and a computer program that transforms an existing program into a program using the present invention by means of an update.

[0098] Furthermore, the computer program unit may be able to provide all the necessary steps of a process for implementing an exemplary embodiment of the method as described above.

[0099] According to another exemplary embodiment of the present invention, there is provided a computer-readable medium, such as a CD-ROM, on which the computer program unit described in the previous part is stored.

[0100] The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium which is provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0101] However, the computer program may also be provided via a network such as the World Wide Web and may be downloaded into the working memory of a data processor from such a network. According to a further exemplary embodiment of the present invention, there is provided a medium for making a computer program unit available for download, the computer program unit being arranged to perform one of the previously described embodiments of the present invention.

[0102] In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality.

Claims

1. An apparatus (130) for processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system, the apparatus comprising: An input unit (132); A point cloud data generator (134); And An output unit (138); Wherein the input unit is configured to receive data collected from the area in the healthcare facility, the data to be sent to the cloud-based processing system, wherein the collected data includes one or more of video data, image data, and ranging data; Wherein the point cloud data generator is configured to process the collected data to generate point cloud data, the point cloud data being capable of being used to provide a point cloud rendering of the area in the healthcare facility; and Wherein the output unit is configured to communicate with a network to transmit the generated point cloud data and corresponding metadata to the cloud-based processing system, wherein the generated point cloud data is preferably capable of being used to determine the presence, location, and / or activity of: (i) one or more persons present in the area in the healthcare facility and / or (ii) one or more objects present in the area in the healthcare facility.

2. The apparatus according to claim 1, Among them, The area in the healthcare facility comprises one or more of the following: Imaging room; Treatment room; Patient preparation room; Patient waiting room; and Patient transfer room.

3. The apparatus according to claim 1 or 2, further comprising: A point cloud data extractor (136); Wherein the point cloud data extractor is configured to: extract a subset of data from the generated point cloud data; and provide the extracted point cloud data to the cloud-based processing system, the subset of data being capable of being used to determine the presence, location, and / or activity of (i) one or more persons present in the area in the healthcare facility, and / or (ii) one or more objects present in the area in the healthcare facility.

4. The apparatus according to claim 3, Among them, The subset of the extracted data comprises one or more of the following: Point cloud data of one or more persons present in an imaging modality room; Point cloud data of one or more objects present in an imaging modality room; One or more data points whose coordinates have changed; And One or more data points that are members of a defined point class.

5. The apparatus according to claim 4, Among them, The one or more objects comprise one or more of the following: scanner table, wearable device, mobile / movable device, and attachable device.

6. The apparatus according to any one of claims 3 to 5, Among them, The point cloud data extractor is configured to adjust one or more of the resolution, size, and sampling rate of the generated point cloud data based on a stage of a clinical workflow, and provide the adjusted point cloud data to the cloud-based processing system.

7. The apparatus according to any one of claims 3 to 6, Among them, The point cloud data extractor is configured to: adjust one or more of the resolution, size, and sampling rate of the generated point cloud data based on processing requirements at the cloud-based processing system; and provide the adjusted point cloud data to the cloud-based processing system.

8. A cloud-based processing system (150) comprising: at least one communication interface (152); and an analysis engine (154); wherein the at least one communication interface is configured to communicate with a network to receive point cloud data providing a point cloud rendering of an area in a healthcare facility; and wherein the analysis engine is configured to apply at least one data analysis algorithm to determine the presence, location, and / or activity of: (i) one or more individuals present in the area in the healthcare facility, and / or / (ii) one or more objects present in the area in the healthcare facility based on the received point cloud data.

9. The cloud-based processing system according to claim 8, Among them, wherein the at least one data analysis algorithm includes a self-learning algorithm configured to continuously learn based on one or more of the received point cloud data, workflow data, device log data, workflow-related data, and user feedback.

10. The cloud-based processing system according to claim 8 or 9, Among them, wherein the analysis engine is configured to apply (i) at least one data analysis algorithm or (ii) a rule-based model and / or a symmetry-based model to identify and fill in missing data.

11. The cloud-based processing system according to any one of claims 8 to 10, further comprising: a data storage device (156); wherein the data storage device is configured to store the received point cloud data and metadata.

12. A system (100) comprising: a sensor device (120) including one or more sensors configured to collect one or more of video data, image data, and ranging data from an area in a healthcare facility; the device (130) according to any one of claims 1 to 7, and the cloud-based processing system (150) according to any one of claims 8 to 11.

13. A method (400) for processing data collected from an area in a healthcare facility for transmission to a cloud-based processing system for processing at the cloud-based processing system, the method comprising: receiving (410) data collected from the area in the healthcare facility to be sent to the cloud-based processing system, wherein the collected data includes one or more of video data, image data, and ranging data; processing (420) the collected data to generate point cloud data capable of providing a point cloud rendering of the area in the healthcare facility; and Communicate with the network (430) to transmit the generated point cloud data and corresponding metadata to the cloud-based processing system, wherein the generated point cloud data can preferably be used to determine the presence, location, and / or activity of: (i) one or more individuals present in the area of the healthcare facility and / or (ii) one or more objects present in the area of the healthcare facility.

14. A computer program product comprising instructions that, when the program is run by a processing unit, cause the processing unit to perform the steps of the method according to claim 13.

15. A computer-readable medium having stored thereon the computer program product according to claim 14.

Images