Sensing device and implementation method for medical facilities

Abstract

A medical system can utilize modular and scalable sensing devices to derive a two-dimensional (2D) or three-dimensional (3D) human body model of a patient in real time based on an image of the patient captured by a sensor such as a digital camera. The 2D or 3D human body model can be visually presented on one or more devices of the medical system and used to facilitate the health care services provided to the patient. In an example, the 2D or 3D human body model can be used to improve the speed, accuracy, and consistency of patient positioning for medical procedures. In an example, the 2D or 3D human body model can be used to perform a unified analysis of the patient's physical condition by associating other medical scan images of the patient with the help of the 2D or 3D human body model. In an example, the 2D or 3D human body model can be used to facilitate surgical navigation, patient monitoring, process automation, etc.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Provisional U.S. Patent Application No. 62 / 941,203 filed on November 27, 2019 and Provisional U.S. Patent Application No. 16 / 860,901 filed on April 28, 2020, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This application is generally in the medical field. Background Art

[0004] The quality of health care services provided to patients depends largely on the amount of information available to the patient and to the medical experts, such as doctors, who provide the services. For example, with respect to radiation therapy and medical imaging, success often depends on the ability to place and maintain the patient in a desired position so that the treatment or scan can be performed in a precise and accurate manner. Having up-to-date knowledge about the patient's physical characteristics (e.g., height, body shape, posture, etc.) in these cases can provide many benefits, including, for example, faster and more accurate positioning of the patient according to a scan or treatment plan, less manual labor, more consistent results, etc. Knowledge about the patient's physical characteristics can also be used to aggregate the patient's different medical records (e.g., based on the patient's common attributes present in the medical records and / or the patient's identity), and derive an overview of the patient's diagnosis and treatment history.

[0005] In other example situations, such as during surgery, information about the patient's size, movement, and / or position can provide insight and guidance for treatment planning and execution. This information can be used, for example, to locate and navigate around the patient's treatment site. When presented visually in real time, this information can also provide a means for monitoring the patient's status during surgery.

[0006] As technologies in fields such as computer vision and artificial intelligence advance, it is desirable to utilize these technologies to acquire patient information in an automatic and real-time manner, and to utilize the acquired information to improve the quality of healthcare services. Summary of the invention

[0007] This article describes systems, methods, and devices for providing health care services to patients using a medical system. The medical system may include a modular and scalable sensing device that can generate a two-dimensional (2D) or three-dimensional (3D) human body model and its representation of the patient in real time based on at least one image of the patient captured by one or more sensors. The sensing device may include or be coupled to these sensors. The sensing device may include one or more processors configured to receive at least one image of the patient from the sensor. In response to receiving at least one image, the sensing device (e.g., one or more processors of a transmitting device) may analyze the image to extract multiple features representing one or more anatomical features of the patient, and estimate the patient's 2D or 3D human body model based on these features. The patient's 2D or 3D human body model may include, for example, a parameterized human body model, and the representation of the 2D or 3D model may include a 2D or 3D mesh representation of the patient.

[0008] Once generated, the 2D or 3D human body model and / or representation thereof of the patient can be transmitted to one or more other devices of the medical system, for example, together with at least one image of the patient captured by one or more sensors. The 2D or 3D human body model and / or representation thereof can be used to improve one or more aspects of health care services provided to the patient, including, for example, patient positioning, patient monitoring, medical scan image unification and analysis, surgical navigation, etc.

[0009] The one or more sensors described herein may include a digital camera, a red, green, and blue (RGB) sensor, a depth sensor, an RGB plus depth (RGB-D) sensor, a thermal sensor such as an infrared (FIR) or near infrared (NIR) sensor, etc. It can be seen that at least one image of the patient captured by the sensor may include a photograph of the patient captured by the digital camera or an RGB or thermal image captured by the corresponding sensor.

[0010] The sensing device can be modular and expandable, and can include one or more slots, each slot including at least one of a power connector or a communication interface circuit. Each slot can be configured to accommodate a corresponding set of sensors or processors, which are configured to work with a corresponding imaging mode or a corresponding patient. The communication interface circuit can be configured to transmit or receive information on behalf of the corresponding sensor or processor accommodated in the corresponding slot. The power connector can be configured to supply power to the corresponding group of sensors or processors accommodated in the corresponding slot. It can be seen that the sensing device can, for example, work with different types of downstream devices and / or different application requirements by increasing or decreasing the number of sensors or processors included in the sensing device.

[0011] The sensing device can be calibrated with other devices in the medical system. For example, one or more processors of the sensing device can be configured to determine the spatial relationship between a first coordinate system associated with the sensing device and a second coordinate system associated with a medical device such as a medical scanner. In this way, a representation of a 2D or 3D human body model of a patient generated by the sensing device can be used with medical scan images obtained from other imaging modes (for example, to align the medical scan images with the representation of the human body model), thereby enabling unified analysis of the patient's medical records obtained from different sources. The 2D or 3D human body model or its representation can also be used to improve the speed, accuracy, and consistency of patient positioning associated with medical procedures to facilitate surgical navigation and patient monitoring, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Examples disclosed herein may be understood in more detail from the following description given by way of example in conjunction with the accompanying drawings.

[0013] Figure 1 is a simplified block diagram illustrating an example medical system described herein.

[0014] Figure 2 is a simplified diagram illustrating how the sensing devices described herein may facilitate the operation of a medical system.

[0015] Figure 3 is a simplified block diagram illustrating an example sensing device described herein.

[0016] Figure 4 is a flow chart illustrating the operation of the sensing device described herein.

[0017] Figure 5 is a flow chart illustrating the operation of the medical system described herein. DETAILED DESCRIPTION

[0018] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0019] Figure 11 is a diagram illustrating an example system 100 for providing health care services in a medical environment or at a medical facility, such as in a hospital. The health care services may include, for example, a medical scan or imaging procedure performed using a medical scanner 102 (e.g., a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) machine, a positron emission tomography (PET) scanner, an X-ray machine, etc.), a radiation therapy procedure delivered by a medical linear accelerator (LINAC) (not shown), or a surgical procedure performed in an operating room. The system 100 may include at least one sensing device 104 (e.g., an image capture device) configured to capture an image of a patient 106 (or object 106) in or around a medical environment (e.g., in front of or inside a medical scanner 102, on a scanning couch or treatment couch, etc.). In an example, the sensing device 104 may be configured to detect the presence of the patient 106 using a sensor such as a motion, radar, or light sensor, and to capture an image of the patient in response to detecting the patient. In an example, the sensing device 104 may not be configured to automatically detect the presence of the patient 106, but may be controlled (e.g., manually controlled, remotely controlled, programmed, etc.) to capture an image of the patient, for example, in response to receiving a control signal instructing the sensing device to do so.

[0020] In an example, the sensing device 104 may include one or more sensors, including one or more cameras (e.g., digital color cameras, 3D cameras, etc.), one or more red, green, and blue (RGB) sensors, one or more depth sensors, one or more RGB plus depth (RGB-D) sensors, one or more thermal sensors (such as infrared (FIR) or near infrared (NIR) sensors), one or more motion sensors, one or more radar sensors, and / or other types of image capture circuitry configured to generate images (e.g., 2D images or photographs) of a person, object, or scene. Depending on the type of sensor or image capture circuitry used, the images generated by the sensing device 104 may include, for example, one or more photographs of a patient taken by a camera, one or more thermal images of a patient generated by a thermal sensor, one or more radar images of a patient generated by a radar sensor, etc. The sensors of the sensing device 104 may be configured to capture images of a patient, object, or scene in response to detecting a patient, object, or scene, based on a preconfigured schedule or time interval, or upon receiving a control signal that triggers image capture.

[0021] In an example, the sensing device 104 can be configured to communicatively couple to sensors (e.g., cameras) already present in the medical system 100 (e.g., sensors external to the sensing device 104), and receive and process images captured by these sensors as if the images were captured by internal sensors of the sensing device 104. For example, the medical system 100 may already include sensors configured for monitoring, diagnosis, or treatment purposes. These sensors are capable of capturing images of patients, objects, or scenes present in the medical system 100 in response to detecting a patient, object, or scene, based on a preconfigured schedule or time interval, or upon receiving a control signal that triggers image capture. The sensor may also have the ability to transmit the captured image (e.g., raw image data including pixel information) to a receiving device in the medical system 100, for example, via a communication interface. In at least these scenarios, the sensing device 104 can be added to the medical system 100, for example, as a receiving device for images generated by the sensor, and obtain information from the image, which can be used to improve the quality of services provided by the medical system 100, as described herein. The sensing device 104 may transmit information obtained from the image (eg, in the form of a 2D or 3D human body model or representation thereof) to a downstream device or application along with, for example, the image data originally received from the sensor.

[0022] The sensing device 104 can be installed or placed in various locations of the medical system 100, such as inside a scanning room, inside a treatment room, inside an operating room, around a registration table, in a hallway, on a medical scanner 102, on a ceiling, near a doorway, on a wall, etc. From these locations, the sensing device 104 can capture images of patients, objects, or scenes in the field of view (FOV) of the sensing device (e.g., from a certain viewpoint or perspective). The FOV of the sensing device 104 can be adjusted (e.g., manually or automatically by sending a control signal to the sensing device) so that multiple images can be taken from different viewpoints or perspectives.

[0023] In an example, the sensing device 104 may be a portable or mobile device, in which case the sensing device may be placed or housed in the various locations described above (e.g., placed on a shelf, attached to a hanging mount, etc.) and may be moved from one location to another. In an example, the sensing device 104 may be physically connected to a location of the medical system 100 or another device (e.g., wired thereto), in which case the sensing device may operate as an integral part of the connected device. And although in Figure 1 Only one sensing device 104 is depicted in the figure, but the medical system 100 may include multiple sensing devices, each sensing device is located at a corresponding position and has a corresponding FOV.

[0024] The sensing device 104 may include a functional unit configured to process images generated by a sensor described herein (e.g., a sensor included in the sensing device and / or a pre-existing sensor external to the sensing device). The functional unit may be coupled to the sensor (e.g., via a wired or wireless communication link) and configured to receive images from the sensor (e.g., via a push mechanism). The functional unit may also be configured to retrieve images from the sensor (e.g., via a pull mechanism), for example, periodically or in response to receiving a control signal that instructs the functional unit to retrieve the image. In an example, the functional unit may be configured to receive a notification from the sensor when an image becomes available and retrieve the image in response to receiving the notification.

[0025] The images captured by the sensor may include two-dimensional (2D) or three-dimensional (3D) images depicting patients, objects, or scenes present in a medical environment. Each of the 2D or 3D images may include a plurality of pixels, lines, and / or vertices. The functional unit may be configured to analyze the images (e.g., at a pixel level) and, for example, use a neural network (e.g., a convolutional neural network) to generate a 2D or 3D model (e.g., a parametric model, such as a parametric model based on a skinned multi-person linear (SMPL) model) of the patient, object, or scene depicted in the image. The 2D or 3D model generated by the functional unit may include one or more 2D key points, one or more 3D key points, one or more parameters for constructing the model (e.g., a set of 72 body shape and / or posture parameters), and / or other information related to the 2D or 3D representation of the patient, object, or scene. The 2D or 3D model may be represented by one or more of, for example, a 2D mesh, a 3D mesh, a 2D outline, a 3D outline, etc., to indicate the patient's posture, body shape, and / or other anatomical features, thereby facilitating multiple downstream medical applications and services for the patient, including, for example, patient positioning, medical plan design, unified or related diagnosis and treatment, medical environment monitoring, surgical navigation, etc. For ease of description, when a 2D or 3D human body model of a patient or a 2D or 3D model of an object or scene is mentioned herein, it should be interpreted to include not only the model itself, but also a representation of the model in any graphical or visual form.

[0026] In an example, the sensing device 104 can act as an edge device (e.g., with limited computing and / or storage capacity) and can pass one or more computing and / or storage tasks (e.g., all computing and / or storage tasks) to a server device. The server device can be a network-based (e.g., cloud-based) server device and can be configured to provide the computing results (e.g., the restored human body 3D mesh model) to other devices of the medical system 100 including the sensing device 104 when completing the computing tasks requested by the sensing device 104 for further processing and / or delivery.

[0027] The sensing device 104 may include a communication circuit configured to exchange information with one or more other devices of the medical system 100, for example, via a communication network 108. The communication network 108 may be a wired or wireless network, or a combination thereof. For example, the communication network 108 may be established on a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi network, etc.), a cellular network (e.g., a long-term evolution (LTE) or a 5G network), a frame relay network, a virtual private network (VPN), a satellite network, and / or a telephone network. The communication network 108 may include one or more network access points. For example, the communication network 108 may include a wired and / or wireless network access point, such as a base station and / or an Internet exchange point, through which one or more components of the medical system 100 may be connected to exchange data and / or other information. Such an exchange may utilize a router, a hub, a switch, a server computer, and / or any combination thereof.

[0028] The communication circuitry of the sensing device 104 may be configured to receive image data generated by the sensors described herein, and may transmit one or more data streams to a receiving device, for example, directly from the sensing device 104 and / or via an edge server, the data streams carrying a 2D or 3D human model of a patient or object generated by the sensing device. In addition to the 2D or 3D model, one or more data streams may also include raw image data (e.g., a raw image including information pixels) generated by the sensor. The raw image data may include, for example, RGB data generated by an RGB sensor, depth data generated by a depth sensor, RGB plus depth (RGB-D) data from an RGB-D sensor, infrared (IR) data from an IR sensor, and the like. The raw image data may be used by downstream devices or applications for various purposes, including, for example, verifying, reconstructing, or refining a 2D or 3D human model generated by the sensing device, comparing or combining the image data with other medical images or scans, and the like.

[0029] The medical system 100 may include a programming device 110 configured to configure and / or control one or more of the medical scanner 102 or the sensing device 104. For example, the programming device 110 may be configured to initialize and modify one or more execution parameters of the medical scanner 102 or the sensing device 104, such as the resolution of captured images, the frequency for exchanging data to and from the sensing device 104 (e.g., the frequency for image transmission or retrieval), the frame rate or bit rate associated with the data exchange, the duration of data storage on the sensing device, etc. The programming device 110 may also be configured to control one or more aspects of the operation of the medical scanner 102 or the sensing device 104, including, for example, triggering a calibration operation for the device, providing calibration parameters to the device (such as calibration parameters related to the spatial relationship between different coordinate systems), adjusting the direction or orientation of the sensor, zooming in or out of the sensor, triggering a system reset, etc. The programming device 110 may include a mobile device, such as a smart phone, a tablet, or a wearable device. The programming device 110 may include a desktop computer, a laptop computer, etc. The programming device 110 may be configured to communicate with the medical scanner 102 and / or the sensing device 104 via the communication network 108. The programming device 110 may receive information and / or instructions from a user (eg, via a user interface implemented on the programming device) and send the received information and / or instructions to the medical scanner 102 or sensing device 104 via the communications network 108 .

[0030] The medical system 100 may also include a processing device 112 configured to receive one or more data streams from the sensing device 104 and / or a device that performs computing or storage tasks on behalf of the sensing device 104. The processing device can be co-located with the sensing device 104 (e.g., in the same room) or remotely located from the sensing device 104 (e.g., in a control room or processing center remote from where the sensing device 104 is located). The processing device 112 can be communicatively coupled to other devices in the medical system 100 including the sensing device 104. The processing device 112 may include a general-purpose computer, a special-purpose computer, one or more computing and storage units, a cloud-based platform, and the like. By way of example only, the cloud-based platforms mentioned herein may include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, inter-clouds, multi-clouds, and the like. Further, although the processing device 112 is Figure 1 102 is depicted as a device separate from the medical scanner 102, but the applicability of the examples provided herein is not limited to this configuration. For example, the processing device 112 can be part of the medical scanner 102 (e.g., included in the medical scanner 102) and still be able to perform the various functions described herein.

[0031] The one or more data streams received by the processing device 112 may include a 2D or 3D model of a patient, object, or scene generated by the sensing device 104 and / or image data (e.g., raw or initial image data) generated by the sensor. As described herein, the 2D or 3D model may be generated by the sensing device 104 or a server device that performs computing or storage tasks on behalf of the sensing device 104. Once received, the 2D or 3D model may be used by the processing device 112 to assist in various aspects of health care services. For example, the processing device 112 may also be coupled to a repository 114 that is configured to store patient medical records, including basic patient information, medical scan images of the patient obtained at different times by other imaging modalities of the medical system 100 (e.g., CT, MR, X-ray, SPECT, PET, etc.), the patient's diagnosis and / or treatment history, etc. The processing device 112 may retrieve all or a subset of the patient's medical records from the repository 114 and analyze the retrieved medical records in conjunction with the 2D or 3D model of the patient provided by the sensing device 104. The processing device 112 may also receive (e.g., in real time) a medical scan image of the patient generated by the medical scanner 102, and analyze the received medical scan image in conjunction with a 2D or 3D model of the patient provided by the sensing device 104. For example, the processing device 112 may receive a medical scan image of the patient from the repository 114 or the medical scanner 102, align the medical scan image with a 2D or 3D human body model of the patient, and visually render (e.g., in an overlaid image) the aligned image and the 2D or 3D model to allow the medical scan image to be presented and analyzed with reference to the anatomical features (e.g., body shape and / or posture) of the patient indicated by the model. In this way, more insight into the organs or tissues of the patient captured in the medical scan image may be obtained based on the additional information provided by the 2D or 3D model. For example, the 2D or 3D model may indicate the patient's posture when the medical scan image was taken.

[0032] For example, the alignment described above can be accomplished by identifying and matching corresponding anatomical landmarks (e.g., joint locations) in the medical scan image and the 2D or 3D model. For example, the alignment can also be accomplished by determining a correlation (e.g., a spatial relationship) between a first coordinate system associated with the 2D or 3D model (e.g., with the sensing device 104) and a second coordinate system associated with the medical environment or medical device, and using the correlation to match the medical scan image and corresponding areas of the 2D or 3D model. For example, the correlation between the two coordinate systems can be determined during system setup based on markers placed in the medical environment and / or by comparing sample images generated by the sensing device 104 and a related medical device (such as a medical scanner 102).

[0033] Using the 2D or 3D model as a common reference, the processing device 112 can align multiple different medical scan images of the patient (e.g., from respective imaging modalities) together, for example, by aligning each medical scan image with the 2D or 3D model, and thereafter aligning one medical scan image with another medical scan image using the 2D or 3D model as an intermediate reference. When referred to herein, alignment of two or more medical scan images or alignment of a medical scan image with a 2D or 3D model can include overlaying one medical scan image with another medical scan image or overlaying a 2D or 3D model with a medical scan image.

[0034] By establishing a correlation between the 2D or 3D human body model of the patient generated by the sensing device 104 and the medical scan images of the patient obtained from other sources or modes, the processing device 112 can determine the target scan or treatment area of ​​the patient and indicate the target area to the patient or a medical professional managing a medical procedure for the patient. For example, the processing device can determine that an organ of the patient (e.g., lungs, head, etc.) needs to be further scanned or treated based on pre-existing medical scan images obtained from other sources or modes. The processing device 112 can align one or more pre-existing medical scan images with the 2D or 3D model of the patient generated by the sensing device 104 (e.g., a 3D mesh of the patient) and determine the location of the organ (e.g., the target scan area) on the 3D mesh. The processing device 112 can visually indicate the determined target area to the patient or medical professional, for example, by marking the target area on the 3D mesh. In this way, positioning of the target scanning or treatment site can be accomplished in real time (e.g., while the patient is waiting to be scanned or treated), non-invasively and / or without the use of radioactive tracers, and the results can be used to provide guidance for treatment planning, program design, and / or surgical navigation (e.g., to guide a surgical robot such as a built-in multi-degree-of-freedom (MDOF) robot or an MDOF robotic arm to avoid collisions, etc.).

[0035] Based on the 2D or 3D human body model of the patient generated by the sensing device 104, the processing device 112 may also be configured to identify background objects in the medical scan image of the patient, and segment or subtract the background objects from the medical scan image so that the medical scan image can be analyzed without interference. For example, a medical scan image of the patient received by the processing device 112 (e.g., from the medical scanner 102) may include a scanning bed in the background of the medical scan image. In order to segment or subtract the area or pixel of the medical scan image corresponding to the scanning bed, the processing device 112 may determine the outline of the patient based on the 2D or 3D human body model of the patient generated by the sensing device 102 while the patient is lying on the scanning bed, and identify the area or pixel of the medical scan image that is outside the outline as the area or pixel corresponding to the scanning bed. The processing device 112 may then segment or subtract the area or pixel corresponding to the scanning bed from the medical scan image, and present the image resulting from the segmentation or subtraction for further analysis.

[0036] The processing device 112 may also utilize a 2D or 3D human body model of the patient generated by the sensing device 104 to facilitate positioning the patient during the scanning or treatment process. For example, as described herein, the 2D or 3D human body model of the patient may include a 3D mesh or outline of the patient and / or parameters indicating multiple anatomical features of the patient (e.g., the patient's body shape, the patient's posture, and / or the patient's distance from the medical scanner 102, etc.). Based on these indications, the processing device 112 may determine, for example, in real time (e.g., while the patient is still in the position or posture indicated by the 2D or 3D human body model) whether the patient's position or posture meets the predetermined scheme for the scanning or treatment process. If the processing device determines that the patient's position or posture does not meet the predetermined scheme, the processing device 112 may provide instructions (e.g., commands) for correcting the patient's position or posture to the patient and / or the medical expert managing the process. The instructions may include visual and / or audio instructions, including, for example, animations and / or demonstrations suitable for augmented reality (AR) devices. In this way, the patient and / or the medical expert can visualize the difference between the patient's current posture and the desired posture (e.g., reference posture) and make necessary adjustments to minimize or eliminate the difference.

[0037] The patient's anatomical features indicated by the 2D or 3D human body model can also be used to determine the patient's radiation consumption level. For example, the processing device 112 can estimate the patient's height and / or weight and the distance between the patient and the medical scanner 102 based on the patient's 2D or 3D human body model, and use this estimate together with the performance parameters of the medical scanner (e.g., the strength of the magnetic field generated by the scanner, the amount of energy released by the scanner, etc.) to calculate the amount of radiation received by the patient. The processing device 112 can generate a report of the radiation level and / or provide an alarm when the level exceeds a certain threshold.

[0038] The 2D or 3D human body model of the patient generated by the sensing device 102 can also be used to determine the identity of the patient. For example, the processing device 112 may include or be coupled to a feature database including known visual features of one or more patients (e.g., key points indicating joint positions, joint angles, facial features, body shape, position, posture, walking pattern, etc.). These known visual features or key points can be pre-calculated and stored in the feature database. In response to receiving the 2D or 3D human body model of the patient from the sensing device 102, the processing device 112 can match the visual features or key points included in the 2D or 3D human body model with the visual features or key points stored in the feature database, and determine the identity of the patient based on the matching score (e.g., a higher score indicates a better match).

[0039] Additionally or alternatively, the processing device 112 may utilize an artificial neural network trained for visual recognition to determine the identity of the patient. In an example, the neural network may include a convolutional neural network (CNN) that includes cascaded layers, each layer being trained to make pattern matching decisions based on a corresponding level of abstraction of visual features contained in a set of images (e.g., in pixels of the images). The training of the neural network may be performed using a large amount of image data and / or a specific loss function, through which the neural network can learn to extract features (e.g., in the form of feature vectors) from a newly provided input image, determine whether the features match features of a known person, and indicate a match result at the output of the neural network. Using one or more such neural networks, the processing device 112 may be configured to compare visual features of a patient indicated by or extracted from a 2D or 3D human body model with visual features extracted from other images of the patient, and determine the identity of the patient based on the comparison or match.

[0040] Once determined, the patient's identity can be used to personalize the healthcare services provided to the patient. For example, processing device 112 can use the patient's identity to retrieve a medical record associated with the patient from repository 114, unify or synthesize the retrieved medical record with other information obtained about the patient (e.g., the patient's body and / or anatomical features indicated by a 2D or 3D human body model), and present the unified or synthesized medical record to allow for a more comprehensive review or analysis of the patient's physical condition.

[0041] As described herein, the processing device 112 can be communicatively coupled to other medical devices (e.g., the medical scanner 102) of the medical system 100. As can be seen, the processing device 112 can be configured to control one or more other medical devices based on information collected from the 2D or 3D human body model of the patient generated by the sensing device 104. For example, when determining the patient's physical characteristics based on the 2D or 3D human body model of the patient, the processing device 112 can transmit a command or control signal (e.g., to the medical scanner 102 or another device that controls the medical scanner 102) to adjust the performance parameters of the medical scanner 102, for example, to better accommodate the patient. Such performance parameters can be associated with, for example, the height of the scanning bed, the scanning angle, the dose level, the position or orientation of the medical scanner, etc., and the command or control signal can be transmitted in digital and / or analog form.

[0042] Further, although examples are provided herein for generating a 2D or 3D human body model of a patient and using the model to improve the quality of health care services for the patient, it should be understood that the sensing device 104 can receive images of multiple patients and, based on these received images, for example, simultaneously generate corresponding 2D or 3D human body models (e.g., representations of the 2D or 3D models) of the patient. The images of the patient can be captured by the same sensor or different sensors, and the generation of the 2D or 3D human body model can be performed by the same functional unit or different functional units. The sensing device 104 can be configured to be modular and / or scalable so that sensors and / or processors (e.g., GPUs) can be added to or removed from the sensing device to adapt to changing application requirements. The modularity and scalability of the sensing device will be described in more detail below.

[0043] The sensing device 104 may also be configured to receive images of objects and scenes present in a medical environment, analyze the images using the techniques described herein, and generate a 2D or 3D model of the object or scene or a report on the object or scene that may indicate the condition of the medical environment. For example, based on the images of one or more objects captured by the sensor, the sensing device 104 may determine the corresponding categories of one or more objects and the number of items in each category, where the category may include at least one of a medical tool, a drug, or a food supply. The sensing device 104 may transmit information (e.g., in a report or visual representation) to the processing device 112, which may use the information to automate or improve the efficiency of facility management (such as inventory management, tool tracking, traffic control, etc.). In an example, the information obtained via the sensing device 104 may indicate conditions in an operating room (e.g., the status of a surgical tool or device, the amount of blood loss of a patient, etc.), so that critical issues may be identified and resolved. In an example, the information obtained via the sensing device 104 may indicate that food or medical supplies at a location are insufficient in stock, so that replenishment may be ordered. In an example, information obtained via the sensing device 104 may indicate heavy traffic in an area of ​​the medical environment, such that patients and medical staff may be notified to avoid the area.

[0044] The medical system 100 and / or the processing device 112 may include a display device 116 and / or an audio device (not shown), which may be configured to display various commands, instructions, alarms, and / or reports generated by the processing device 112 and / or other devices in the system, as described herein. The display device 116 may include one or more monitors (e.g., a computer monitor, a TV monitor, a tablet computer, a mobile device such as a smart phone, etc.), one or more speakers, one or more augmented reality (AR) devices (e.g., AR goggles), and / or other accessories configured to facilitate auditory or visual presentation. The display device 116 may be communicatively coupled to the processing device 112 and / or the sensing device 104 via the communication network 108 or another suitable communication link. As described herein, the information or instructions presented via the display device 116 may include the desired position and posture of the patient for the medical procedure, the posture taken by the patient during the past scan, the adjustment instructions for the patient to enter the desired position or posture, the surgical navigation instructions, the level of dose consumption, etc. The information and / or instructions may be presented to the patient 106 in various formats, including, for example, audio, video, animation, AR presentation, etc.

[0045] The systems, methods, and devices described herein may be used by Figure 2 It is further shown that Figure 2 It shows where a sensing device (e.g., Figure 11. The scanning room 200 may be equipped with a medical scanner 202 (e.g., a vertical X-ray scanner) as shown in FIG. Figure 1 The medical scanner 102 shown) and the display device 216 (e.g., Figure 1 The display device 216 shown in FIG. 1 is a display device 116. The display device 216 may be communicatively coupled to the medical scanner 202 (eg, via a wired or wireless communication network) or may be part of the medical scanner 202. Figure 1 The sensing device 104 of the sensing device ( Figure 2 A 2D or 3D human body model (not shown) can be added to the scanning room 200 (e.g., mounted on a wall, attached to a hanging mount under the ceiling, placed on a shelf, etc.) and is configured to generate a 2D or 3D human body model, such as a 3D mesh, of the patient 206 while the patient is standing in front of the medical scanner 101. As described herein, the 2D or 3D human body model can be generated based on one or more images (e.g., photographs or thermal images) of the patient captured by a sensor included in the sensing device (e.g., a camera, a thermal sensor, etc.) or by one or more sensors already installed in the scanning room 202 (e.g., existing sensors outside the sensing device). A functional unit of the sensing device (e.g., which can include one or more processors) can receive these images of the patient, analyze the images to determine a plurality of features representing one or more anatomical features of the patient (e.g., joint positions, joint angles, etc.), and estimate a 2D or 3D human body model of the patient based on the plurality of features. The 2D or 3D human body model can indicate one or more physical features of the patient, such as the patient's body shape and / or posture, the patient's position relative to the medical scanner 202, and / or other anatomical features of the patient.

[0046] The 2D or 3D human body model may be estimated and transmitted in one or more data streams in real time (e.g., while the patient is standing in front of the medical scanner 101) to a processing device (e.g., Figure 1The processing device 112 shown in the figure can be located inside the scanning room 200 or remote from the scanning room 200. The transmission can be performed, for example, via communication circuitry of the sensing device, and in addition to the 2D or 3D human body model, one or more data streams can also include an image of the patient generated by the sensor (e.g., raw image data such as pixel data). In response to receiving the 2D or 3D human body model generated by the sensing device and / or the image generated by the sensor, the processing device can render the 2D or 3D human body model and / or image of the patient in a manner that allows the information to be used to improve various aspects of health care services provided to the patient 206, including, for example, unified or coordinated medical diagnosis, guided imaging or surgical procedures, patient positioning and / or monitoring, medical facility management, etc. For example, the processing device can overlay the image of the patient with a 2D or 3D human body model of the patient (e.g., a 3D mesh), such as Figure 2 As shown, a target scanning site of a patient can be determined, and a medical scanner can be controlled (e.g., remotely and / or automatically) to focus on the target scanning site (e.g., to collimate an X-ray beam toward the target scanning site to reduce radiation, improve subject contrast and image quality, etc.). As another example, a 2D or 3D human body model can be used to identify, detect and / or track the movement and activity of a patient or subject in a medical environment for purposes such as process monitoring, process optimization, resource pre-allocation, resource utilization analysis, automated process recording, workflow analysis and optimization, automated process cost code estimation, etc.

[0047] Figure 3 An example sensing device 300 (eg, Figure 1 The sensing device 300 may include a sensor 302, a functional unit 304, and a power supply, which are configured to be housed in a housing. Although two sensors are shown in the figure, it should be understood that the sensing device 300 may include any number of sensors, or the sensing device 300 may not include any sensors and may instead be configured to receive images from an external sensor (e.g., an existing sensor in a medical system). Further, although in Figure 3 One or more components are shown as being inside or outside the functional unit 304, but these components are not limited to the configuration shown in the figure and can be moved inside or outside the functional unit 304 without affecting the function of the sensing device described herein.

[0048] As described herein, sensor 302 may include an RGB sensor, a depth sensor, an RGB plus depth (RGB-D) sensor, a thermal sensor (such as a FIR or NIR sensor), a radar sensor, a motion sensor, a camera (e.g., a digital camera), and / or other types of image capture circuitry configured to generate images (e.g., 2D images or photographs) of people, objects, and / or scenes in the sensor's FOV. And the images generated by sensor 302 may include, for example, one or more photographs, thermal images, and / or radar images of a person, object, or scene. Each image may include a plurality of pixels that collectively represent a graphical view of a person, object, or scene and that may be analyzed to extract features representing one or more characteristics of the person, object, or scene.

[0049] The sensor 302 may be communicatively coupled to the functional unit 304, for example, via a wired or wireless communication link. In an example, the sensor 302 may be configured to transmit images generated by the sensor to the functional unit 304 (e.g., via a push mechanism). In an example, the functional unit 304 may be configured to retrieve images from the sensor 302 (e.g., via a pull mechanism). The transmission and / or retrieval may be performed periodically (e.g., based on a preconfigured schedule) or in response to receiving a control signal that triggers the transmission or retrieval. Such a control signal may be sent, for example, by the sensor 302 (e.g., when an image has become available), or by a remote control device such as a mobile device or a system controller (e.g., when input is received from a user).

[0050] The sensor 302 may be configured to receive one or more control signals (e.g., digital control messages) from the functional unit 304 that may affect the operation of the sensor 302. For example, the sensor 302 may receive a command from the functional unit 304 to adjust the FOV of the sensor (e.g., by manipulating the direction or orientation of the sensor). As another example, the sensor 302 may receive a command from the functional unit 304 that changes the resolution at which the sensor captures images of a person, object, or scene.

[0051] The sensor 302 and / or the functional unit 304 (e.g., one or more components of the functional unit 304) can be powered by a power source 306, which can include an alternating current (AC) power source or a direct current (DC) power source (e.g., a battery power source). When a DC power source such as a battery power source is used, the power source 306 can be rechargeable, for example, by receiving a charging current from an external source via a wired or wireless connection. For example, the charging current can be received by connecting the sensing device 300 to an AC outlet via a charging cable and / or a charging adapter (including a USB adapter). As another example, the charging current can be received wirelessly by placing the sensing device 300 in contact with a charging pad.

[0052] Functional unit 304 may include one or more of communication interface circuit 308, data processing unit 310, computing unit 312, data rendering unit 314, memory 316, or programming and / or calibration application programming interface (API) 318. Note that Figure 3 The illustrated architecture is provided only as an example and is not intended to limit the scope of the present disclosure to such an architecture. Figure 3 The exact components shown. Two or more of the components (e.g., the functionality of the components) may be combined, any of the components may be divided into sub-components, any of the components may be omitted, more components may be added, etc. It can be seen that even though the functions of the sensing device 300 are described herein as being associated with a corresponding one or more components, it should be understood that these functions may also be performed by different components and / or divided among multiple other components.

[0053] exist Figure 3 In the example shown, the functional unit 304 can be configured to receive or retrieve an image of the patient from the sensor 302 via a communication interface circuit 308, which can include one or more wired and / or wireless network interface cards (NICs), such as an Ethernet card, a WiFi adapter, a mobile broadband device (e.g., a 4G / LTE / 5G card or chipset), etc. In an example, a respective NIC can be designated to communicate with a respective sensor. In an example, a unified NIC can be designated to communicate with multiple sensors.

[0054] The images received or retrieved from the sensor 302 may be provided to the data processing unit 310, which may be configured to analyze the images and estimate (e.g., construct or restore) a model based on the images to depict (e.g., mathematically and / or visually) one or more characteristics (e.g., body shape, posture, etc.) of the patient depicted in the image. For example, the data processing unit 310 may be configured to analyze at least one of the images generated by the sensor 302 (e.g., at a pixel level), identify a plurality of features representing one or more anatomical or physical features of the patient, and estimate parameters (e.g., mesh parameters) that may be used to construct a human body model (e.g., a parameterized human body model) of the patient. In an example, the plurality of features may represent joint positions and / or joint angles of the patient as depicted in at least one image generated by the sensor 302, and the human body model may include an SMPL model defined by a plurality of parameters indicating one or more characteristics of the patient. The parameters estimated by the data processing unit 310 may include one or more body shape parameters β and / or one or more posture parameters θ. The body shape parameters may include coefficients of a principal component analysis (PCA) space that may be used to determine (e.g., restore) a mixed body shape of the patient. The posture parameters can be derived based on the positions and / or angles of multiple joints of the patient (e.g., 23 joints included in the skeletal equipment and the root joint) restored from at least one image, and can indicate the posture of the patient. Based on the body shape and / or posture parameters (e.g., a set of 72 parameters corresponding to 23 joints), the data processing unit 310 can determine multiple mesh vertices, which can be used to generate a representation of a 2D or 3D human body model built for the patient (e.g., a 3D mesh representation). In addition, the data processing unit 310 can also be configured to determine the identity of the patient using the body shape / posture parameters and / or features extracted from at least one image. The data processing unit 310 can indicate the identity of the patient to other devices or components of the medical system to personalize the health care services provided to the patient.

[0055] The functionality of the data processing unit 310 can be facilitated by a computing unit 312, which can be configured to perform various computationally intensive tasks, such as feature extraction and / or feature classification, based on images generated by the sensor 302. The computing unit 312 can include one or more neural networks trained for visual recognition, such as one or more convolutional neural networks (CNNs) and / or one or more deep neural networks (DNNs). The neural network can include multiple layers (e.g., an input layer, one or more convolutional layers, one or more non-linear activation layers, one or more pooling layers, one or more fully connected layers, and / or an output layer). Each layer can correspond to a plurality of filters (e.g., kernels), and each filter can be designed to detect a set of key points that collectively represent a corresponding visual feature or pattern. The filter can be associated with a corresponding weight that, when applied to the input, produces an output indicating whether certain visual features or patterns have been detected. The weights associated with the filters can be learned by the neural network through a training process that includes: inputting a large number of images from one or more training data sets to the neural network (e.g., in a forward pass), calculating the loss caused by the weights currently assigned to the filters (e.g., based on a loss function, such as a margin-based loss function), and updating (e.g., in a backward pass) the weights assigned to the filters so as to minimize the loss (e.g., based on stochastic gradient descent). Once trained, the neural network can take an image at an input layer, extract and / or classify visual features or patterns from the image, and provide an indication at an output layer as to whether the extracted features match known features and / or whether the extracted features fall into a particular category or classification.

[0056] In addition to or in lieu of the neural network described above, the computing unit 312 may include or be coupled to a feature database configured to store a plurality of known features of the patient (e.g., facial features, body shape, body contours, joint positions, joint angles, walking patterns, postures, etc.). Each feature may correspond to a combination of key points arranged in a particular manner in the image, such as a point where the direction of a boundary of an object changes abruptly, an intersection between two or more edge segments, etc. Key points may be characterized by a well-defined position in the image space and / or stability to illumination or brightness perturbations. Therefore, these key points may be identified based on image derivatives, edge detection, curvature analysis, etc. And once identified, the key points and / or features represented by the key points may be described by feature descriptors or feature vectors. In an example embodiment of such feature descriptors or vectors, information related to the features (e.g., the appearance of the local neighborhood of each key point) may be represented by (e.g., encoded into) a series of numerical values ​​stored in the feature descriptor or vector. The descriptor or vector may then be used as a "fingerprint" to distinguish one feature from another, or to match one feature to another.

[0057] One or more human models (e.g., parameters for constructing the human model) generated by the data processing unit 310 and / or the computing unit 312 can be provided to the data rendering unit 314, which can be configured to generate a representation of the human model (e.g., a 2D or 3D representation, such as a 2D or 3D mesh representation) that depicts one or more anatomical or physical features of the patient. For example, the data rendering unit 314 can receive a plurality of mesh vertices determined by the data processing unit 310 and / or the computing unit 312. Each vertex can include respective position, normal, texture, and / or shading information. Based on these vertices, the data rendering unit 314 can create a 2D or 3D mesh of the patient, for example, by connecting a plurality of vertices with edges to form polygons (e.g., triangles), connecting a plurality of polygons to form surfaces, using a plurality of surfaces to determine a 3D shape, and applying textures and / or shading to the surfaces and / or shapes. Once created, the 2D or 3D representation can be output by the data rendering unit 314 in one or more data streams, for example, via the communication interface circuit 308 to a receiving device (e.g., Figure 1 Further, in addition to the human body model information generated by the data processing unit 310 and / or the computing unit 312, the data rendering unit 314 may also be configured to receive raw image data generated by the sensor 302 and output the raw image data to the receiving device, for example, in a data stream that is the same as or different from the data stream carrying the representation.

[0058] Each of the data processing unit 310, the computing unit 312, or the data rendering unit 314 may include one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a reduced instruction set computer (RISC) processor, an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or a combination thereof. The data processing unit 310, the computing unit 312, and / or the data rendering unit 314 may also include other types of circuits or processors capable of performing the functions described herein. Further, the data processing unit 310, the computing unit 312, or the data rendering unit 314 may utilize the memory 316 to facilitate one or more of the operations described herein. For example, the memory 316 may include a machine-readable medium configured to store data and / or instructions, which, when executed, causes the processing unit 310, the computing unit 312, or the data rendering unit 314 to perform one or more of the functions described herein. Examples of machine-readable media may include volatile or nonvolatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Figure 3 Although not shown, the sensing device 300 may also include one or more large-capacity storage devices, including disks, such as built-in hard disks, removable disks, magneto-optical disks, CD-ROMs or DVD-ROM disks, etc., on which instructions and / or data may be stored to facilitate the execution of the functions described herein.

[0059] The sensing device 300 and / or the functional unit 304 can be configured to be modular and extensible, so that sensors, communication circuits, data processing units, computing units, and / or data rendering units can be added to or removed from the sensing device 300, for example, to accommodate different system settings, configurations, and / or requirements in a medical environment (e.g., the medical system 100). For example, if output quality is a priority in a medical environment, a high-resolution sensor (e.g., a high-resolution camera) can be included (e.g., added) in the sensing device 300 to meet the priority. On the other hand, if the priority is about output speed (e.g., frame rate), a sensor with a lower resolution (e.g., a camera) and / or a communication circuit with a faster bit rate (e.g., an Ethernet card instead of a WiFi card) can be used to meet the output requirements. As another example, the sensing device 300 can be configured to work (e.g., work simultaneously) with multiple devices in a medical environment (such as multiple imaging modes (e.g., CT, MR, etc.), in which case the sensing device can include sensors, communication circuits, power supplies, processors (e.g., the data processing units, computing units, and / or data rendering units described herein) for corresponding groups of corresponding medical devices. As another example, the sensing device 300 can be configured to receive images of multiple patients (e.g., from different sensors) and, based on the images, for example, simultaneously generate corresponding 2D or 3D models of the patients. In such a scenario, the sensing device can include a corresponding set of sensors, communication circuits, power supplies, processors (e.g., data processing units, computing units, and / or data rendering units described herein) for capturing and processing corresponding images of corresponding patients.

[0060] In an example, the sensing device 300 and / or the functional unit 304 may include a plurality of slots (e.g., expansion boards, etc.), each slot being equipped with at least one of a power connector or a communication circuit (e.g., a network interface card, a USB port, etc.) capable of transmitting and receiving information via a wired or wireless communication link. Sensors and / or processors (e.g., data processing units, computing units, and / or data rendering units described herein) may be accommodated (e.g., inserted) in these slots, where they may receive power via corresponding power connectors and perform data exchange with one or more internal or external devices via corresponding communication circuits. These sensors and processors may have capabilities similar to the sensors 302, data processing units 310, computing units 312, and data rendering units 314 described herein, respectively, and may be added to or removed from the sensing device 300, for example, to adapt to changing conditions and / or requirements in the medical environment in which the sensing device 300 is installed.

[0061] For example, the sensing device 300 may include a first set of one or more sensors configured to capture an image of a first patient (e.g., a patient undergoing an X-ray scan) and a first set of one or more processors (e.g., a data processing unit 310, a computing unit 312, and a data rendering unit 314) configured to generate a first 2D or 3D model of the first patient and provide the model to a first receiving device (e.g., a controller associated with an X-ray scanner). The first set of one or more sensors and / or the first set of one or more processors may be housed in a first slot of the sensing device 300, which may provide power and / or communication services to the sensors and / or processors. The sensing device 300 may also include a second slot configured to house (e.g., provide power and / or communication services to) a second set of one or more sensors and / or a second set of one or more processors (e.g., units similar to the data processing unit 310, the computing unit 312, and the data rendering unit 314). This second set of sensors can be configured to capture images of a second patient (e.g., a patient undergoing a CT scan), and the second set of one or more processors can be configured to generate a second 2D or 3D model of the second patient and provide the model to a second receiving device (e.g., a controller associated with a CT scanner). In this way, the sensing device 300 can be modular and scalable to handle data processing tasks associated with different patients and / or imaging modalities. In other example situations, such as when the computational, communication, and / or data storage workload approaches or exceeds the capabilities of a set of sensors and / or processors, more sensors and / or processors can be added to share the workload.

[0062] The operation of the sensing device 300 may be performed using, for example, Figure 1 The remote programming device of the programming device 110 in the programming device 110 is configured and / or controlled through the programming / calibration API 318. In an example, the programming / calibration API 318 can be configured to receive commands (e.g., one or more digital messages) from the programming device that change the execution parameters of the sensing device 300, such as the orientation and / or FOV of the sensor, the resolution of the image captured by the sensor, the quality required for the representation of the 2D or 3D model of the patient generated by the sensing device, the period of receiving or retrieving images from the sensor, the bit rate at which the sensing device transmits the 2D or 3D human body model of the patient and / or the raw image data captured by the sensor, etc. In response to receiving the command from the programming device, the sensing device 300 (e.g., the functional unit 304) can adjust one or more aspects of its operation according to the command. For example, if the command specifies a higher output quality, the sensing device 300 can output a high-resolution grid in response, and if the command specifies a higher frame rate, the sensing device 300 can output a lower resolution grid, but at an increased frame rate.

[0063] The sensing device 300 (e.g., the functional unit 304) may also be configured to receive ad hoc commands via the programming / calibration API 318. Such ad hoc commands may include, for example, a command to zoom in or out of the sensor, a command to reset the sensing device 300 (e.g., restart the device or reset one or more execution parameters of the device to default values), a command for the sensing device 300 to transmit or retransmit certain types of data such as metadata generated by the sensing device and related to the human body mesh (e.g., estimated parameters used to construct the human body mesh) to a receiving device, a command to enable or disable a specific function of the sensing device 300 (such as whether the sensing device should attempt to determine the identity of the patient), etc. The sensing device 300 (e.g., the functional unit 304) may also be programmed and / or trained (e.g., over a network) via the programming / calibration API 318. For example, the sensing device 300 may receive training data and / or operating logic via the programming / calibration API 318 during and / or after an initial configuration process.

[0064] The sensing device 300 (e.g., the functional unit 304) can be calibrated with the medical environment in which the sensing device is installed and / or with one or more other devices in the medical environment (such as the medical scanner 102 in the medical system 100). For example, calibration can be performed during initial configuration of the sensing device 300 and / or in response to receiving a calibration command via the programming / calibration API 318. Calibration can include determining a first coordinate system associated with the sensing device 300 and a coordinate system associated with the medical environment or a medical device in the medical environment (such as Figure 1 In an example, the sensing device 300 (e.g., the functional unit 304) may be configured to determine (e.g., learn) a spatial relationship between a first coordinate system and a second coordinate system associated with a medical scanner 102 in the medical scanner. In an example, the sensing device 300 (e.g., the functional unit 304) may be configured to determine (e.g., learn) a spatial relationship between a first coordinate system and a second coordinate system based on an offset between respective origins of the two coordinate systems and / or a rotation angle between respective X-axes or Y-axes of the two coordinate systems. The sensing device 300 may receive information about such an offset and / or rotation angle via a configuration message (e.g., transmitted by a programming device). In an example, the sensing device 300 may be configured to learn the offset and / or rotation angle by comparing a sample image generated by the sensing device and the medical scanner, for example, based on an annotation or marked area of ​​the image. In an example, the sensing device 300 may be configured to learn the offset and / or rotation angle (e.g., during a calibration process) based on markers placed in a medical environment (e.g., such as one or more objects placed in a corner of a scanning room).

[0065] Once the spatial relationship (e.g., spatial correlation) between the first coordinate system and the second coordinate system is determined, the sensing device 300 and / or other devices in the medical system can use the spatial relationship to perform human body model restoration, medical scan image analysis, etc. For example, the sensing device 300 can receive an image of a patient from a sensor (e.g., a camera), the image including a scanning bed in the image background, and the sensing device 300 can have knowledge about the position of the scanning bed in the scanning room, as defined by the coordinate system of the scanning room (e.g., the sensing device can have learned or been given the position of the scanning bed during system configuration). If the sensing device 300 can determine the spatial relationship between the coordinate system of the scanning room and the coordinate system of the sensor that captured the image, the sensing device 300 can, for example, use a transformation matrix, based on the offset and / or rotation angles described herein, etc. to convert the position of the scanning bed in the former coordinate system (e.g., associated with the scanning room) to the position in the latter coordinate system (e.g., associated with the sensing device or sensor). The sensing device 300 can then segment the scanning bed from the image so that a 2D or 3D model can be generated for only the patient (e.g., excluding the scanning bed from the model).

[0066] A first coordinate system associated with the sensing device 300 (eg, a sensor of the sensing device) and a first coordinate system associated with the medical device (eg, Figure 1 The spatial relationship (e.g., spatial correlation) between the second coordinate system associated with the medical scanner 102 and the medical scanner 102 can also be used by a processing device or controller associated with the medical device to process or analyze the medical information collected for the patient via the medical device together with the 2D or 3D human body model of the patient generated by the sensing device 300. For example, based on the spatial correlation between the respective coordinate systems associated with the sensing device 300 and the medical scanner 102, the processing device or controller associated with the medical scanner 102 can project a medical scan image of the patient captured by the medical scanner 102 onto the 2D or 3D human body model of the patient generated by the sensing device 300 to allow for unified analysis of the medical scan image, as described herein.

[0067] Although a description of the sensing device 300 is provided using an image of a patient as an example, it should be understood that the sensing device 300 can also use similar techniques to process images of objects or scenes. As described herein, information extracted from images of objects or scenes can be used for various facility management purposes in a medical environment, including, for example, inventory management, tool tracking, traffic control, facility monitoring, etc.

[0068] Figure 4 is an example of a sensor device (e.g., Figure 1 The sensing device 104 or Figure 3Flowchart of example operations performed by a sensing device 300 in . The sensing device can be configured to periodically (e.g., based on a predetermined time interval, upon receiving a command to start the operation, or in response to detecting a patient in a medical environment) start the example operation at 402. At 404, the sensing device can receive one or more images of a patient captured by a sensor such as an RGB sensor, a thermal sensor, or a digital camera. As described herein, the sensor can be an existing sensor in a medical environment (e.g., an existing camera) or can be a sensor included in the sensing device. At 406, the sensing device can analyze the received images and extract a plurality of features representing one or more anatomical features of the patient as depicted in the one or more images. The features can represent, for example, joint positions and / or joint angles of the patient.

[0069] Based on the extracted features, at 408, the sensing device can determine (e.g., estimate) a set of parameters related to the human body model of the patient, for example, by restoring the body shape of the patient and / or multiple joint angles or positions of the patient based on the extracted features. The set of parameters can include (e.g.,) one or more body shape parameters that collectively indicate the body shape of the patient and one or more posture parameters that collectively indicate the posture of the patient. Using the body shape and / or posture parameters (e.g., a set of 72 parameters corresponding to 23 joints of the patient), the sensing device can create a representation of the human body model (e.g., a 2D or 3D mesh representation) at 410, for example, by determining multiple vertices of a mesh associated with the human body model and using these vertices to create a mesh. At 412, the sensing device can transmit the human body model (e.g., mesh) and / or image received at 404 to a receiving device. Then, the set of operations of the sensing device can end at 414.

[0070] For simplicity of explanation, the operation of the sensing device is described in Figure 4 1 and described in a particular order herein. However, it should be understood that these operations can occur in various orders, simultaneously and / or with other operations not presented or described herein. In addition, not all illustrated operations need to be performed by the sensing device.

[0071] Figure 5 is an example of a method that can be used by the medical system described herein (e.g., Figure 1Flowchart of example operations performed by one or more devices of a medical system 100 in the medical environment. Operations may be started periodically at 502. At 504, an image of a patient may be captured in a medical environment, for example, by a sensing device (e.g., sensing device 100 or 300) or an existing sensor (e.g., an existing camera) in the medical environment. At 506, a human body model may be derived based on the image of the patient captured at 504. As described herein, such a human body model may be derived by a sensing device and may be represented by a 2D mesh, a 3D mesh, a 2D outline, a 3D outline, etc., which indicates one or more anatomical or physical features (e.g., body shape, posture, etc.) of the patient as depicted in the captured image.

[0072] At 508, a determination may be made as to whether the patient needs to be positioned for the upcoming medical procedure. If it is determined that the patient needs to be positioned, a further determination may be made at 510 based on the derived human body model as to whether the current position of the patient indicated by the human body model meets the requirements of the protocol designed for the medical procedure. If the current position of the patient meets the requirements, a confirmation may be provided to the patient and / or a medical professional overseeing the medical procedure at 512. Otherwise, at 512, adjustment instructions (e.g., commands) may be provided to the patient to help the patient move to the desired position. Adjustment instructions (e.g., control signals) may also be provided to a medical device involved in the process to change one or more relevant execution parameters of the device (e.g., the height of the scanning bed).

[0073] After the operation at 512, or if it is determined at 508 that the patient does not need to be positioned, another determination can be made at 514 as to whether there are medical scan images of the patient that need to be analyzed. If it is determined that there are medical scan images to be analyzed, then at 516, the medical scan images can be analyzed using the human body model as a reference. For example, one or more of the medical scan images can be aligned with the human body model and / or with each other based on common anatomical landmarks identified in the medical scan images and the human body model. The aligned medical scan images can then be analyzed together to obtain a holistic view of the patient's physical condition.

[0074] After the operation at 516, or if it is determined at 514 that there is no medical scan image to be analyzed, another determination may be made at 518 as to whether a target region for scanning or treatment needs to be located. If such a target region exists, a human body model may be used to locate the region and / or provide navigation guidance toward the target region, for example, based on body structure information included in the human body model. After the operation at 520, or if it is determined at 518 that there is no target region to be located, the set of operations may end at 522.

[0075] For simplicity of explanation, the operation of the medical system is Figure 5 , and described herein in a particular order. However, it should be understood that these operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. In addition, it should be noted that Figure 5 Not all operations that the medical system can perform are depicted and not described herein. It should also be noted that not all illustrated operations need to be performed by the medical system.

[0076] Further, the medical environments described herein may include fitness or rehabilitation facilities, and the sensing devices may be used to monitor and / or guide physical therapy / rehabilitation, training, exercise, etc. For example, the sensing devices may be used in these settings to track the movement of a patient or athlete (e.g., in real time), compare the movement to guidelines / instructions, and recommend necessary adjustments to improve training or rehabilitation activities.

[0077] Although the present disclosure has been described according to certain embodiments and generally associated methods, the changes and transformations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit the present disclosure. Other changes, substitutions and variations are also possible without departing from the spirit and scope of the present disclosure. In addition, unless otherwise specifically stated, discussions using terms such as "analyze," "determine," "enable," "identify," "modify," etc. refer to the actions and processes of a computer system or similar electronic computing device, which manipulate and transform data represented as physical (e.g., electronic) quantities within registers and memories of a computer system into other data represented as physical quantities within a computer system memory or other such information storage, transmission or display device.

[0078] It should be understood that the above description is intended to be illustrative, rather than restrictive. After reading and understanding the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present disclosure should be determined with reference to the appended claims and the full scope of equivalents to which such claims are assigned.

Claims

1. A sensing device for use in a medical environment, the sensing device comprising: a first set of one or more sensors configured to capture at least one image of a first patient while the first patient is present in the medical environment; A first set of one or more processors configured to: analyzing the at least one image of the first patient captured by the first set of one or more sensors to extract a plurality of first features representing one or more anatomical features of the first patient as depicted by the at least one image of the first patient; estimating a first two-dimensional (2D) or three-dimensional (3D) human body model of the first patient based on the plurality of first features extracted from the at least one image of the first patient, wherein the first 2D or 3D human body model is estimated by restoring a plurality of first joint angles or positions of the first patient as depicted in the at least one image of the first patient or a posture of the first patient as depicted in the at least one image of the first patient; and generating a first representation of the estimated first 2D or 3D human body model; and a first communication circuit configured to transmit the first estimated representation of the first 2D or 3D human body model to a first receiving device so that a command is generated by the first receiving device based on the first estimated representation of the first 2D or 3D human body model to guide or alert the first patient or another person, or to control a medical device in the medical environment; wherein the 2D or 3D human body model of the first patient comprises a parameterized human body model of the first patient, and the first representation of the estimated first 2D or 3D human body model comprises a 2D or 3D mesh representation, wherein the features extracted from the at least one image of the first patient are expressed using corresponding feature vectors, and the one or more anatomical features of the first patient comprise at least one of a joint position or a joint angle of the first patient.

2. The sensing device according to claim 1, wherein the first group of one or more processors is further configured to align the medical scan image and the first 2D or 3D human body model, and render the aligned medical scan image and the first 2D or 3D human body model, wherein The medical scan image comes from a repository or a medical scanner.

3. The sensing device according to claim 1, wherein: The first set of one or more processors is configured to analyze the at least one image of the first patient or estimate the first 2D or 3D body model of the first patient using at least one trained convolutional neural network.

4. The sensing device according to claim 1, further comprising a first slot and a second slot, wherein: The first slot is configured to accommodate the first set of one or more sensors or the first set of one or more processors; The second slot is configured to accommodate a second set of one or more sensors or a second set of one or more processors; The second set of one or more sensors is configured to capture at least one image of a second patient while the second patient is present in the medical environment; and The second set of one or more processors is configured to: analyzing the at least one image of the second patient captured by the second set of one or more sensors to extract a plurality of second features representing one or more anatomical features of the second patient as depicted by the at least one image of the second patient; estimating a second 2D or 3D human body model of the second patient based on the plurality of second features extracted from the at least one image of the second patient, wherein the second 2D or 3D human body model is estimated by restoring a plurality of second joint angles or positions of the second patient as depicted in the at least one image of the second patient or a posture of the second patient as depicted in the at least one image of the second patient; and A second representation of the estimated second 2D or 3D human body model is generated.

5. The sensing device according to claim 4, wherein: The first slot includes at least one of the first communication circuit or the first power connector, the first power connector is configured to power the first group of one or more sensors or the first group of one or more processors, and wherein the second slot includes at least one of the second communication circuit or the second power connector, the second communication circuit is configured to transmit the second representation of the second 2D or 3D human body model to a second receiving device, the second power connector is configured to power the second group of one or more sensors or the second group of one or more processors, wherein the first receiving device is associated with a first imaging mode of the medical environment, and the second receiving device is associated with a second imaging mode of the medical environment, each of the first imaging mode or the second imaging mode is associated with at least one of a computed tomography (CT) scanner, a magnetic resonance (MR) scanner, or an X-ray scanner.

6. The sensing device according to claim 1, wherein: The first representation of the estimated first 2D or 3D human body model of the first patient is used by the first receiving device to determine a posture of the first patient relative to the medical device, and wherein the command generated by the first receiving device is associated with adjusting the posture of the first patient relative to the medical device or adjusting an execution parameter of the medical device based on the posture of the first patient.

7. The sensing device according to claim 1, wherein: The first group of one or more processors is also configured to determine a spatial relationship between a first coordinate system and a second coordinate system, the first coordinate system being associated with the first group of one or more sensors, and the second coordinate system being associated with the medical environment, wherein the first group of one or more processors is configured to determine the spatial relationship between the first coordinate system and the second coordinate system based on information received from a remote programming device, or wherein the first group of one or more processors is configured to analyze the at least one image of the first patient captured by the first group of one or more sensors, including: the first group of one or more processors is configured to segment background objects from the at least one image of the first patient based on the spatial relationship between the first coordinate system and the second coordinate system.

8. The sensing device according to claim 1, wherein: The first set of one or more processors is further configured to receive one or more execution parameters from a remote programming device, the one or more execution parameters associated with estimating the first 2D or 3D human body model or generating the first representation of the estimated first 2D or 3D human body model.

9. A sensing device for use in a medical environment, the sensing device comprising: a communication circuit configured to receive at least one image of the patient from the sensor; and One or more processors configured to: analyzing the at least one image of the patient received from the sensor to extract a plurality of features representing one or more anatomical features of the patient as depicted by the at least one image of the patient; estimating a two-dimensional (2D) or three-dimensional (3D) human body model of the patient based on the plurality of features extracted from the at least one image of the patient, wherein the 2D or 3D human body model is estimated by restoring a plurality of joint angles or positions of the patient as depicted in the at least one image of the patient or a posture of the patient as depicted in the at least one image of the patient; generating a representation of the estimated 2D or 3D human body model; and transmitting the estimated 2D or 3D human body model to a receiving device via the communication circuit; wherein the 2D or 3D human body model of the patient comprises a parameterized human body model of the patient, and the representation of the estimated 2D or 3D human body model comprises a 2D or 3D mesh representation, wherein the features extracted from the at least one image of the patient are expressed using corresponding feature vectors, and wherein the one or more anatomical features of the patient comprise at least one of a joint position or a joint angle of the patient.

10. A method implemented in a medical system, the method comprising: receiving at least one image of the patient from a sensor; analyzing the at least one image of the patient received from the sensor to extract a plurality of features representing one or more anatomical features of the patient as depicted by the at least one image of the patient; estimating a two-dimensional (2D) or three-dimensional (3D) human body model of the patient based on the plurality of features extracted from the at least one image of the patient, wherein the 2D or 3D human body model is estimated by restoring a plurality of joint angles or positions of the patient as depicted in the at least one image of the patient or a posture of the patient as depicted in the at least one image of the patient; generating a representation of the estimated 2D or 3D human body model; and transmitting the estimated 2D or 3D human body model to a receiving device via a communication circuit; wherein the 2D or 3D human body model of the patient comprises a parameterized human body model of the patient, and the representation of the estimated 2D or 3D human body model comprises a 2D or 3D mesh representation, wherein the features extracted from the at least one image of the patient are expressed using corresponding feature vectors, and wherein the one or more anatomical features of the patient comprise at least one of a joint position or a joint angle of the patient.

Images