Subject tracking method for medical imaging and medical imaging system

Abstract

The embodiment of the invention provides a subject tracking method for medical imaging, a computer readable storage medium and a medical imaging system. The method comprises the following steps: acquiring multi-frame image data which is shot by a camera device and contains a subject; detecting information of anatomical key points in each frame of image data in the multiple frames of image data; and determining the state of the subject according to the information of the anatomical key points.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of medical devices, and in particular, to a method for tracking a subject for medical imaging and a medical imaging system. Background Art

[0002] There are various modes for performing medical imaging of a subject, including but not limited to a computed tomography (CT) system, a magnetic resonance imaging (MRI) system, etc. When the subject undergoes the above-mentioned medical imaging scan, the subject needs to enter the scan room, and the operator needs to enter the operation room to control the medical imaging system. Therefore, the operator cannot timely monitor the position or posture of the subject in the scan room, so that the operator cannot timely start or stop the scan-related processing, and even cannot timely discover the safety accidents that occur to the subject. Summary of the Invention

[0003] The embodiments of the present application provide a method for tracking a subject for medical imaging and a medical imaging system.

[0004] According to one aspect of the embodiments of the present application, a method for tracking a subject for medical imaging is provided, and the method includes:

[0005] Obtaining multiple frames of image data including the subject captured by an imaging device;

[0006] Detecting information of anatomical key points in each frame of the multiple frames of image data;

[0007] Determining the state of the subject according to the information of the anatomical key points.

[0008] According to one aspect of the embodiments of the present application, a computer-readable storage medium is provided, and the computer-readable storage medium includes a stored computer program, wherein when the computer program is run, the method for tracking a subject for medical imaging described in the previous aspect is executed.

[0009] According to one aspect of the embodiments of the present application, a medical imaging system is provided, and the system includes:

[0010] An imaging device that captures multiple frames of image data including the subject;

[0011] A controller connected to the imaging device and configured to execute the method for tracking a subject described in the previous aspect.

[0012] With reference to the following description and drawings, specific embodiments of the present application are disclosed in detail, indicating the ways in which the principles of the embodiments of the present application can be adopted. It should be understood that the embodiments of the present application are not limited thereby. Within the spirit and scope of the appended claims, the embodiments of the present application include many variations, modifications, and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings included herein are used to provide a further understanding of the embodiments of the present application, which form a part of the specification, and are used to illustrate the embodiments of the present application and, together with the written description, to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other embodiments based on these drawings without creative efforts. In the drawings:

[0014] Figure 1 is a schematic diagram of a magnetic resonance imaging system according to an embodiment of the present application;

[0015] Figure 2 is a schematic diagram of a subject tracking method for medical imaging according to an embodiment of the present application;

[0016] Figure 3 is a schematic diagram of a medical imaging system according to an embodiment of the present application;

[0017] Figure 4 is a schematic diagram of a magnetic resonance imaging system according to an embodiment of the present application;

[0018] Figure 5 is a schematic diagram of a subject tracking device for medical imaging according to an embodiment of the present application;

[0019] Figure 6 is a schematic diagram of a subject tracking method for medical imaging according to an embodiment of the present application;

[0020] Figure 7 is a schematic diagram of a medical imaging method according to an embodiment of the present application;

[0021] Figure 8 is a schematic diagram of anatomical key points according to an embodiment of the present application;

[0022] Figure 9 is a schematic diagram of a user interface 1 according to an embodiment of the present application;

[0023] Figure 10 is a schematic diagram of a scanning method 1 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Referring to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations and equivalents falling within the scope of the appended claims.

[0025] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish different elements in terms of appellation, but do not represent the spatial arrangement or time sequence of these elements, etc. These elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the related listed terms. Terms such as "comprising", "including", "having", etc. mean the presence of the stated features, elements, components or assemblies, but do not exclude the presence or addition of one or more other features, elements, components or assemblies.

[0026] In the embodiments of the present application, the singular forms "a", "the", etc. include the plural forms and should be broadly understood as "a kind of" or "a class of" rather than being limited to the meaning of "one"; in addition, the term "the" should be understood to include both the singular form and the plural form unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least partly according to...", and the term "based on" should be understood as "at least partly based on...", unless the context clearly indicates otherwise.

[0027] Features described and / or illustrated for one embodiment can be used in the same or similar way in one or more other embodiments, combined with the features in other embodiments, or replace the features in other embodiments. The term "comprising / including" as used herein means the presence of features, whole things, steps or components, but does not exclude the presence or addition of one or more other features, whole things, steps or components.

[0028] The medical imaging systems described herein include but are not limited to computed tomography (CT) systems, magnetic resonance imaging (MRI) systems, C-arm imaging systems, positron emission tomography (PET) systems, single photon emission computed tomography (SPECT) systems, ultrasound systems, X-ray imaging systems, or any other suitable medical imaging systems.

[0029] The following takes a magnetic resonance imaging system as an example for illustration, but the embodiments of the present application are not limited thereto.

[0030] For ease of understanding, Figure 1 a magnetic resonance imaging (MRI) system 100 of some embodiments of the present invention is shown.

[0031] The MRI system 100 includes a scanning unit 111. The scanning unit 111 is configured to perform magnetic resonance scanning on an object (e.g., a human body) 170 to generate a reconstructed image of a region of interest of the object 170, and the region of interest may be a pre-determined anatomical site or anatomical tissue.

[0032] The operation of the MRI system 100 is controlled by an operator workstation 110, which includes an input device 114, a control panel 116, and a display 118. The input device 114 may be a joystick, keyboard, mouse, trackball, touch-activated screen, voice control, or any similar or equivalent input device. The control panel 116 may include a keyboard, touch-activated screen, voice control, buttons, sliders, or any similar or equivalent control device. The operator workstation 110 is coupled to and communicates with a computer system 120, which enables the operator to control the generation and viewing of images on the display 118. The computer system 120 includes a plurality of components that communicate with each other via an electrical and / or data connection module 122. The connection module 122 may be a direct wired connection, a fiber optic connection, a wireless communication link, etc. The computer system 120 may include a central processing unit (CPU) 124, a memory 126, and an image processor 128. In some embodiments, the image processor 128 may be replaced by an image processing function implemented in the CPU 124. The computer system 120 may be connected to an archival media device, permanent or backup memory, or a network. The computer system 120 may be coupled to and communicate with a separate MRI system controller 130.

[0033] The MRI system controller 130 includes a set of components that communicate with each other via an electrical and / or data connection module 132. The connection module 132 can be a direct wired connection, a fiber optic connection, a wireless communication link, etc. The MRI system controller 130 can include a CPU 131, a sequence pulse generator 133 that communicates with the operator workstation 110, a transceiver (or RF transceiver) 135, a memory bank 137, and an array processor 139. In some embodiments, the sequence pulse generator 133 can be integrated into the resonance assembly 140 of the scan unit 111 of the MRI system 100. The MRI system controller 130 can receive commands from the operator workstation 110, be coupled to the scan unit 111 to indicate the MRI scan sequence to be executed during an MRI scan, and be used to control the scan unit 111 to execute the process of the above-mentioned magnetic resonance scan. The MRI system controller 130 is also coupled to and communicates with a gradient driver system 150, which is coupled to a gradient coil assembly 142 to generate a magnetic field gradient during an MRI scan.

[0034] The sequence pulse generator 133 may also receive data from a physiological acquisition controller 155, which receives signals from a plurality of different sensors (such as an electrocardiogram (ECG) signal from an electrode attached to a patient), and these sensors are connected to an object or patient 170 undergoing an MRI scan. The sequence pulse generator 133 is coupled to and communicates with a scan room interface system 145, which receives signals from various sensors associated with the state of the resonance assembly 140. The scan room interface system 145 is also coupled to and communicates with a patient positioning system 147, which sends and receives signals to control the movement of the patient table to a desired position for an MRI scan.

[0035] The MRI system controller 130 provides gradient waveforms to the gradient driver system 150, which includes G x (in the x direction), G y (in the y direction), and G z (in the z direction) amplifiers, etc. Each G x , G y , and G zGradient amplifiers each drive a corresponding gradient coil in gradient coil assembly 142 to generate magnetic field gradients for spatially encoding MR signals during an MRI scan. Gradient coil assembly 142 is disposed within resonance assembly 140, which also includes a superconducting magnet having a superconducting coil 144 that, in operation, provides a static, uniform longitudinal magnetic field B0 through a cylindrical imaging volume 146. Resonance assembly 140 also includes an RF body coil 148 that, in operation, provides a transverse magnetic field B1 that is substantially perpendicular to B0 throughout the cylindrical imaging volume 146. Resonance assembly 140 may also include an RF surface coil 149 for imaging different anatomical structures of a patient undergoing an MRI scan. RF body coil 148 and RF surface coil 149 may be configured to operate in a transmit and receive mode, a transmit mode, or a receive mode.

[0036] The x-direction may also be referred to as the frequency encoding direction or k in k-space. x The y-direction may be referred to as the phase encoding direction or k in k-space. y direction. G x may be used for frequency encoding or signal readout and is commonly referred to as the frequency encoding gradient or readout gradient. G y may be used for phase encoding and is commonly referred to as the phase encoding gradient. G z may be used for slice (layer) position selection to obtain k-space data. It should be noted that the slice selection direction, phase encoding direction, and frequency encoding direction may be modified according to actual needs.

[0037] An object or patient 170 undergoing an MRI scan may be positioned within the cylindrical imaging volume 146 of resonance assembly 140. A transceiver 135 in MRI system controller 130 generates RF excitation pulses amplified by RF amplifier 162 and provides them to RF body coil 148 through a transmit / receive switch (T / R switch) 164.

[0038] As described above, RF body coil 148 and RF surface coil 149 may be used to transmit RF excitation pulses and / or receive resulting MR signals from a patient undergoing an MRI scan. MR signals emitted by nuclei excited within the body of a patient undergoing an MRI scan may be sensed and received by RF body coil 148 or RF surface coil 149 and sent back through T / R switch 164 to preamplifier 166. T / R switch 164 may be controlled by a signal from sequence pulse generator 133 to electrically connect RF amplifier 162 to RF body coil 148 during the transmit mode and to connect preamplifier 166 to RF body coil 148 during the receive mode. T / R switch 164 may also enable RF surface coil 149 to be used in the transmit mode or the receive mode.

[0039] In some embodiments, the MR signals sensed and received by RF body coil 148 or RF surface coil 149 and amplified by preamplifier 166 are stored in memory 137 as a raw k-space data array for post-processing. The reconstructed magnetic resonance images can be obtained by transforming / processing the stored raw k-space data.

[0040] In some embodiments, the MR signals sensed and received by RF body coil 148 or RF surface coil 149 and amplified by preamplifier 166 are demodulated, filtered, and digitized in the receive portion of transceiver 135 and transmitted to memory 137 in MRI system controller 130. For each image to be reconstructed, the data is rearranged into separate k-space data arrays, and each of these separate k-space data arrays is input to array processor 139, which is operated to Fourier transform the data into an array of reconstructed images.

[0041] Array processor 139 uses a transform method, most commonly the Fourier transform, to create images from the received MR signals. These images are transferred to computer system 120 and stored in memory 126. In response to commands received from operator workstation 110, the data used to reconstruct the images can be stored in long-term memory or can be further processed by image processor 128 and transferred to operator workstation 110 for presentation on display 118.

[0042] In various embodiments, the components of computer system 120 and MRI system controller 130 can be implemented on the same computer system or on multiple computer systems. It should be understood that Figure 1 the MRI system 100 shown is for illustration. Suitable MRI systems can include more, fewer, and / or different components.

[0043] MRI system controller 130 and image processor 128 can each or jointly include a computer processor and a storage medium on which a program of predetermined data processing to be executed by the computer processor is recorded. For example, programs for implementing scan processing (such as scan procedures, imaging sequences), image reconstruction, image processing, etc. can be stored on the storage medium. For example, a program for implementing the magnetic resonance imaging method of the embodiments of the present invention can be stored. The above storage medium can include, for example, ROM, floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, or non-volatile memory card.

[0044] The above-mentioned "imaging sequence" (hereinafter also referred to as a scanning sequence or a pulse sequence) refers to a combination of pulses with specific amplitudes, widths, directions, and time sequences applied during the execution of magnetic resonance imaging scans. These pulses typically may include, for example, radiofrequency pulses and gradient pulses. The radiofrequency pulses may include, for example, radiofrequency excitation pulses, radiofrequency refocusing pulses, inversion recovery pulses, etc. The gradient pulses may include, for example, the above-mentioned gradient pulses for slice selection, gradient pulses for phase encoding, gradient pulses for frequency encoding, gradient pulses for phase shift (phase offset), gradient pulses for discrete phase (phase dispersion), etc. Generally, multiple scanning sequences can be preset in a magnetic resonance system so that a sequence suitable for clinical detection requirements can be selected. The clinical detection requirements may include, for example, the imaging site, imaging function, imaging effect, etc.

[0045] Currently, in medical imaging systems (such as magnetic resonance imaging systems or computed tomography imaging systems, etc.), it is desirable to introduce a camera device to obtain information on the vision and morphology of the human body. The camera device can be installed near the examination bed to collect auxiliary information of the subject in a non-contact manner to the greatest extent. The acquired images can be used to assist medical imaging operations, such as automatic positioning and automatic identification of the subject's direction, etc.

[0046] However, the inventor found that the subject located in the scanning room may get up and adjust their posture at any time, or leave the scanning bed, or fall off the scanning bed, etc. But after the operator enters the operation room and runs a series of processes such as positioning or identifying the direction of the medical imaging system, it may not be possible to timely monitor changes in the subject's state, resulting in the failure of the aforementioned work process and even accidents.

[0047] In addition, the inventor also found that although there are many tracking methods in the current computer vision field, the above methods are not easily directly applicable to medical imaging systems. For example, in some scenarios, it is necessary to cover the subject's surface with occluding objects (such as coils, blankets, complex clothes, masks, etc.). Since some areas of the subject are covered by occluding objects, the existing tracking methods always lose track of the subject.

[0048] To address at least one of the above technical problems, an embodiment of the present application proposes a method for tracking a subject in medical imaging. The state of the subject is determined based on the anatomical key points identified in the images captured by the imaging device, enabling automatic tracking of the subject after entering the scanning room. The determined state of the subject can be used as a trigger condition for other procedures during the scanning process, such as automatic positioning, automatic orientation recognition, user interface display, execution of occlusion algorithms, etc., to further automate the scanning process and improve scanning efficiency. Additionally, when it is detected that the subject gets up, leaves the examination table, or falls, the currently running programs can be stopped in a timely manner. For example, the programs for automatic positioning, automatic orientation recognition, user interface display, and occlusion algorithms can be stopped to avoid incorrect execution of the programs. Moreover, an alarm can be issued to alert the clinical operator.

[0049] This will be described below in conjunction with embodiments.

[0050] An embodiment of the present application provides a method for tracking a subject in medical imaging. Figure 2 It is a schematic diagram of the method for tracking a subject in medical imaging according to an embodiment of the present application, as Figure 2 shown, the method includes:

[0051] 201, Obtain multiple frames of image data including the subject captured via the imaging device;

[0052] 202, Detect the information of the anatomical key points in each frame of the multiple frames of image data;

[0053] 203, Determine the state of the subject based on the information of the anatomical key points.

[0054] In some embodiments, the imaging device can be used to obtain the image data of the subject, and the image data can include at least one of two-dimensional optical image data and depth image data, but is not limited thereto. For example, the imaging device can be a 3D camera, which can capture the whole body of the subject in real time to generate a video stream. Each frame in the video stream includes the two-dimensional optical image and the depth image. The two-dimensional optical image can be a two-dimensional RGB image, and each pixel value of the depth image reflects the distance between the camera and the corresponding position of the subject. The imaging device can be installed at the ceiling or wall directly above the examination table of the medical imaging system. The embodiments of the present application are not limited thereto.

[0055] The above takes the imaging device as a 3D camera as an example, but the present application is not limited thereto. For example, the above two-dimensional optical image data and depth image data can also be obtained by an independent 2D optical camera and depth camera respectively, and no further examples will be given here.

[0056] In some embodiments, the multi-frame image data may be consecutive multi-frame image data within a predetermined time, or non-consecutive multi-frame image data (for example, there are X frames between adjacent frames). For example, the predetermined time may be 1 minute, 5 minutes, etc. The multi-frame image data may be the image data of the Nth frame, the (N + 1)th frame... the (N + M)th frame in the video stream. That is to say, the number of frames in the multi-frame image data is M, and M is an integer greater than or equal to 2. The embodiments of the present application are not limited thereto.

[0057] In some embodiments, in 202, a deep learning algorithm may be used to detect each frame of image data respectively, and identify the anatomical key points in each frame of image data. The number and type of anatomical key points in each frame of image data may be the same or different. The anatomical key points include at least two of the following: the top of the head, shoulders, nose, eyes, ears, arms, elbows, wrists, hips, knees, heart, pelvis, abdomen, chest, ankles, etc. As Figure 8 shown, anatomical key points 1 to 16 are detected.

[0058] For example, a human pose estimation model may be used to extract the anatomical key points from the image data. The human pose estimation model is implemented based on a deep learning algorithm. Specifically, reference may be made to related technologies. For example, a densepose model, an openpose model, an HRNet model, etc. may be used to identify the above-mentioned anatomical key points from the image data. Among them, the previously collected image data of multiple volunteers may be used as the input parameter set, and the multiple anatomical key points corresponding to each image data calibrated in advance may be used as the output parameter set. The human pose estimation model is trained using the input parameter set and the output parameter set, and the trained human pose estimation model is used to extract the anatomical key points.

[0059] In some embodiments, after the anatomical key points are identified, the information of the anatomical key points is determined. The information of the anatomical key points includes at least one of the following information: the anatomical type of the key point, the position coordinates of the key point, the confidence level of the key point, and the depth information of the key point.

[0060] For example, the position coordinates of the key point may be the coordinates of the anatomical key point in the image coordinate system (two-dimensional pixel coordinates), or the three-dimensional space coordinates of the camera coordinate system or the three-dimensional space coordinates of the medical imaging coordinate system. Regarding coordinate conversion, reference may be made to related technologies and will not be elaborated here.

[0061] For example, the anatomical types of the key points include the aforementioned anatomical parts such as the top of the head, shoulders, nose, eyes, ears, arms, elbows, wrists, hips, knees, heart, pelvis, abdomen, chest, ankles, etc.

[0062] For example, the confidence of a key point can be represented in the form of a probability. For example, it refers to the probability that the true position of the key point is located at the detected position, etc. This application is not limited to this, and the confidence information of the key point can also be in other forms or other meanings.

[0063] For example, the depth information of a key point includes the depth value of the key point, which is represented by the distance value between the pixel position of each key point and a reference plane (such as a camera).

[0064] The above anatomical type, coordinates, and confidence can all be used as the output parameters of the above human pose estimation model. After inputting the two-dimensional optical image data into the human pose estimation model, the human pose estimation model outputs them together. When the human pose estimation model estimates the coordinates of a key point, the depth value of the coordinate position corresponding to the coordinate position in the depth image is determined as the depth information of the key point.

[0065] In some embodiments, for each frame of image data, the information I of the above anatomical key points is determined, and then the information of the anatomical key points of multiple frames of image data is obtained. The information of the anatomical key points can be arranged in chronological order and correspond to each frame of the above image data in sequence. For example, the information of the anatomical key points of the multiple frames of image data can be expressed as:

[0066] I1 N ,I2 N ,I3 N ,I4 N …I A N

[0067] I1 N+1 ,I2 N+1 ,I3 N+1 ,I4 N+1 …I B N+1

[0068] I1 N+2 ,I2 N+2 ,I3 N+2 ,I4 N+2 …I C N+2

[0069] …

[0070] I1 N+M ,I3 N+M ,I5 N+M …I D N+M

[0071] Among them, the subscript numbers of I can represent the types of key points, such as Figure 8As shown, 0 and 1 represent the shoulders, 2 and 3 represent the elbows, 4 and 5 represent the wrists, 6 and 7 represent the pelvis, 8 and 9 represent the knees, 10 and 11 represent the ankles, 12 represents the top of the head, 13 represents the nose, 14 represents the neck, 16 represents the abdomen, 15 and 17 represent the chest. The anatomical key points detected in a frame of image data can be arranged in ascending order of the numbers representing the types of key points. If the information of the corresponding anatomical key points is not included, it means that the corresponding anatomical key points are not detected in this frame of image data. For example, for the (N + M)-th frame of image, the elbows and wrists on the left side of the subject are not detected.

[0072] In some embodiments, the state of the subject can be determined based on all the detected anatomical key points, but the embodiments of the present application do not limit thereto. It is also possible to select the anatomical key points with a confidence level higher than a threshold (e.g., 60%) from the detected anatomical key points; and determine the state of the subject according to the information of the selected anatomical key points, so as to improve the detection accuracy of the subject's state.

[0073] In some embodiments, the states of the subject that can be detected include at least one of whether the subject moves, the moving direction of the subject, the position when there is no movement, whether the subject is partially occluded, and the posture of the subject (getting up, falling, lying down, etc.), which will be described separately below.

[0074] In some embodiments, in 203, the reference position of the subject in each frame of image data can be determined according to the information of the anatomical key points; and according to the reference positions in the multi-frame image data, it can be determined whether the state of the subject is no movement or in movement.

[0075] In this embodiment, for the information of the anatomical key points in each frame of image data, a reference position is determined. The reference position can be the center of gravity position determined based on the anatomical key points, or the center position of the upper limb, or the head position, or the center position of the lower limb, etc., and the embodiments of the present application do not limit thereto. For example, for each frame of image data, calculate the average value of the position coordinates of multiple anatomical key points (all anatomical key points or anatomical key points with a confidence level higher than the threshold, which will not be repeated below), or perform a clustering algorithm on the position coordinates of some anatomical key points (e.g., the clustering algorithm for the anatomical key points of the upper body to calculate the center position of the upper limb), or select the coordinates of the anatomical key points of the head to obtain the image coordinates of the above reference position or the three-dimensional space coordinates in the camera coordinate system or the three-dimensional space coordinates in the medical imaging coordinate system. Arrange the reference positions in chronological order, corresponding to each frame of image data in the above multi-frame image data in turn. For example, for the information I1 of the anatomical key points in the N-th frame of image data N , I2 N , I3 N , I4N …I A N Determine the reference position R N For the information I1 of the anatomical key points of the (N + 1)-th frame of image data N+1 I2 N+1 I3 N+1 I4 N+1 …I B N+1 Determine the reference position R N+1 And so on, for the information I1 of the anatomical key points of the (N + M)-th frame of image data N+M I2 N+M I3 N+M I4 N+M …I D N+M Determine the reference position R N+M The reference positions of the multi-frame image data arranged in chronological order can be expressed as: R N R N+1 R N+2 …R N+M .

[0076] In this embodiment, after determining the reference position of each frame of image data, the reference position can be tracked to determine the change trend of the coordinates of the reference position over time. According to the change trend of the coordinates of the reference position over time, it is determined whether the state of the subject is non-moving or moving. For example, if the coordinates of multiple reference positions arranged in chronological order remain unchanged or the change range is within a preset range, it is determined that the subject is non-moving; if the coordinates of multiple reference positions arranged in chronological order change or the change range exceeds the preset range, it is determined that the subject is moving.

[0077] In some embodiments, optionally, the moving direction of the subject or the position when the subject is non-moving can also be determined according to at least one of the comparison result between the reference position and the reference position in the multi-frame image data or the depth information of the reference position in the multi-frame image data.

[0078] In this embodiment, when it is determined that the subject is non-moving according to the reference position, the reference position in the same coordinate system can be further compared with the reference position (for example, the coordinate position of the bed board carrying the subject). If the reference position is within the range of the reference position, it is determined that the subject has been stably located on the bed board.

[0079] In this embodiment, when it is determined that the subject is moving according to the reference position, according to the coordinate changes of multiple reference positions arranged in chronological order, the moving trajectory of the subject can be obtained. The initial reference position R in chronological order NCompare with the reference position in the same coordinate system (e.g., the coordinate position of the bed board carrying the subject). If the initial reference position is not within the range of the reference position, but its trajectory is gradually towards the reference position, it is determined that the moving direction of the subject is towards the bed board. If the initial reference position is within the range of the reference position, but its trajectory is gradually away from the reference position, it is determined that the moving direction of the subject is away from the bed board.

[0080] In this embodiment, when determining that the subject is moving based on the reference position, the moving direction of the subject can be further determined according to the depth information of the reference position in the multi-frame image data. When obtaining the coordinates of the reference position, determine the depth value of the coordinate position corresponding to the coordinate position in the depth image as the depth information of the reference position. For the depth values of multiple reference positions arranged in chronological order, if it is determined that the depth value of the reference position is getting closer and closer to the depth value of the reference position (e.g., the coordinate position of the bed board carrying the subject), it is determined that the moving direction of the subject is towards the bed board. If it is determined that the difference between the depth value of the reference position and the depth value of the reference position (e.g., the coordinate position of the bed board carrying the subject) is getting larger and larger, it is determined that the moving direction of the subject is away from the bed board. Or, if the depth value of the reference position is getting larger, it is determined that the moving direction of the subject is away from the bed board, and if the depth value of the reference position is getting smaller, it is determined that the moving direction of the subject is towards the bed board.

[0081] In some embodiments, in 203, the state of the subject as to whether it is partially occluded can be determined according to the change in the number and position of the anatomical key points in the multi-frame image data. Among them, for the number of anatomical key points in the multi-frame image data arranged in chronological order, if the number of anatomical key points becomes smaller over time and the coordinates of the remaining anatomical key points that have not decreased remain unchanged (or change within a preset range), it is determined that the state of the subject is being partially occluded. If the number of anatomical key points becomes larger over time and the coordinates of the remaining anatomical key points except for the part of the anatomical key points that have become larger remain unchanged, it is determined that the state of the subject is being removed from partial occlusion. Among them, in order to improve the detection accuracy, the detection of whether it is partially occluded can be performed when it is determined that the subject has no movement and has stably located on the bed board, but the embodiments of the present application do not limit this.

[0082] In some embodiments, in 203, the posture change of the subject can be determined according to at least one of the position change of the anatomical key points, the distance change between the anatomical key points, and the depth change of the anatomical key points in the multi-frame image data. The posture change includes, but is not limited to, getting up or lying down or falling, etc. Here, lying down, falling, and getting up are taken as examples for illustration.

[0083] In this embodiment, the state of the subject as getting up or lying down can be determined according to the change in the distance between anatomical key points. The distance can be the distance in the image coordinate system, the camera coordinate system, or the medical imaging coordinate system. The embodiments of the present application are not limited thereto. For example, according to the coordinates of the head and pelvis among the anatomical key points, the distance between the head and pelvis in the same frame of image is calculated, and the change trend of this distance calculated from the multi-frame image data arranged in chronological order is determined. If the distance becomes smaller over time, it is determined that the subject gets up and changes from the lying position to the sitting position. If the distance becomes larger over time, it is determined that the subject lies down and changes from the sitting position to the lying position.

[0084] In this embodiment, the state of the subject as getting up or lying down can be determined according to the change in the depth of the anatomical key points. For example, the depth information of the head among the anatomical key points is determined, and the change trend of this depth information over time is determined from the depth information of the head in the multi-frame image data arranged in chronological order. If the change trend of the depth information of the head is approaching the imaging device (for example, the depth value becomes smaller), it is determined that the subject gets up and changes from the lying position to the sitting position. If the change trend of the depth information of the head is away from the imaging device (for example, the depth value becomes larger), it is determined that the subject lies down and changes from the sitting position to the lying position.

[0085] In this embodiment, it can be determined whether the state of the subject is a fall according to the change in the position of the anatomical key points in the multi-frame image data. For example, the coordinates of the anatomical key points in the multi-frame image data arranged in chronological order. If the position coordinates of all the anatomical key points used to determine the state of the subject in the multi-frame image data change from being within the area of the bed board position to not being within the area of the bed board position, and after not being within the area of the bed board position, the position coordinates of more than a predetermined proportion (for example, 60%) of the anatomical key points remain unchanged (or change within a preset range), it is determined that the subject has fallen from the bed board.

[0086] The above various detection methods for the state of the subject can be implemented alone or in combination. The embodiments of the present application are not limited thereto. For example, when determining the moving direction of the subject, the state of the subject can be determined by combining the changes in the coordinates and depth values of the reference position. Only when the state of the subject determined according to the changes in the position coordinates and depth values is the same, will the same state be used as the detected state of the subject. Examples are not given one by one here.

[0087] In addition, for multiple sets of M-frame image data in the video stream acquired by the imaging device, at least one of the above determination methods can be respectively executed. For example, after the subject enters the scanning room, first acquire the image data from the Nth frame to the N+Mth frame, determine the subject's state based on the detected anatomical key points, and then acquire the image data from the N+M+1th frame to the N+2Mth frame, determine the subject's state based on the detected anatomical key points, and so on until the subject leaves the scanning room.

[0088] In some embodiments, after determining the state of the subject, at least one of the following is triggered: a display change of the user interface of the medical imaging system, an automatic positioning process, a process of automatically identifying the subject's orientation, a blanket and coil shielding detection algorithm, or an alarm.

[0089] As an example, after determining the state of the subject, a display change of the user interface of the medical imaging system is triggered. For example, when it is determined that the subject's state is moving, or the subject's state is lying down, or partially occluded, or stably located on the bed board, the anatomical key points of the detected subject are automatically triggered to be displayed on the user interface of the medical imaging system display. Figure 9 is a schematic diagram of the user interface of an embodiment of the present application, as Figure 9 shown, the user interface 90 includes a display window 91 for the image captured by the imaging device. When it is determined that the subject's state is lying down on the bed board 912, the real-time state of the captured subject is displayed on the display window 91, and the identified anatomical key points are superimposed and displayed on the subject 911. Optionally, the identified subject state 92, such as "moving in", "moving out", "occluded", etc., can also be superimposed and displayed on the display window 91.

[0090] As an example, after determining the status of the subject, automatic positioning processing and processing for automatically identifying the direction of the subject can be triggered. For example, when it is determined that the subject is stably located on the bed board, the medical imaging system is automatically triggered to start positioning and identify the direction of the subject (whether the head is facing the scanning center or the feet are facing the scanning center). This positioning processing and the processing for identifying the direction of the subject can be related technologies, and the embodiments of the present application are not limited thereto. For example, for the positioning processing, the imaging device acquires an image containing the subject, detects the coordinate position (in the pixel coordinate system) of the part to be examined in the image, and based on the type and coordinate position of the part to be examined, uses a deep learning algorithm or a machine learning algorithm to estimate the distance Z (in the medical imaging system coordinate system) that the part to be examined needs to move to align with the scanning center. Among them, the types and coordinate positions of the parts to be examined of multiple volunteers located on the bed board can be pre-acquired as an input parameter set, and the moving distances corresponding to the types and coordinates of each part to be examined, which are pre-measured or calculated, are used as an output parameter set. The deep learning algorithm is trained using the input parameter set and the output parameter set, and the trained deep learning algorithm is used to determine the distance Z that the part to be examined needs to move. For the identification of the direction of the subject, the image acquired by the imaging device is also used, and an image recognition algorithm is used to identify the direction of the subject, which will not be elaborated here one by one.

[0091] Figure 10 is a schematic diagram of a scanning method according to an embodiment of the present application, as Figure 10 shown, the method includes:

[0092] 1001, acquiring multiple frame image data containing the subject, and determining the status of the subject according to the information of the detected anatomical key points;

[0093] 1002, when it is determined that the subject is stably located on the bed board, acquiring an image of the subject located on the bed;

[0094] 1003, identifying the coordinate position of the part to be examined in the image;

[0095] 1004, determining the distance Z that the bed board needs to move according to the coordinate position;

[0096] 1005, identifying the direction of the subject according to the image;

[0097] 1006, controlling the bed board to move a distance Z to complete the positioning of the part to be examined;

[0098] 1007, determining the slice position to be scanned according to the identified direction of the subject and the part to be examined, and performing a scan according to the set scan parameters.

[0099] The above embodiments can appropriately adjust the execution order between various operations. In addition, some other operations can be added or some of the operations can be reduced. The present application is not limited thereto.

[0100] As an example, when it is determined that the state of the subject is partially occluded, the occlusion algorithm (detecting the blanket and the coil) is automatically triggered for execution; for the occlusion algorithm, the image obtained by the imaging device is also used, and at least one of the information of the color channels and the depth information of the image is used, and the body parts covered by the blanket or the coil are determined by using a deep learning algorithm. For details, reference can be made to related technologies and will not be elaborated here one by one. Optionally, after the occlusion algorithm is executed, the position of the key points estimated by using the deep learning algorithm can also be superimposed and displayed at the occluded position of the subject on the display window 91.

[0101] As an example, when it is determined that the state of the subject is moving and the moving direction is moving away from the bed board or it is determined that the subject gets up, the anatomical key points of the subject are no longer displayed on the user interface of the medical imaging system display, the positioning and the subject direction recognition are stopped, or the occlusion algorithm is stopped from being executed. For example, after the subject is stabilized on the bed board, the automatic positioning and the subject direction recognition process, and the execution of the occlusion algorithm, etc. are triggered. However, due to the occurrence of situations such as the discomfort of the subject, the subject may get up to adjust the posture or even leave the bed board and leave the scanning room. When the above states are detected, the positioning and the subject direction recognition are stopped, or the occlusion algorithm is stopped from being executed to avoid the wrong execution of the program.

[0102] As an example, when it is determined that the state of the subject is falling, an alarm is issued (a sound alarm or an alarm is displayed on the user interface 90), the anatomical key points of the subject are no longer displayed on the user interface of the medical imaging system display, the positioning and the subject direction recognition are stopped, or the occlusion algorithm is stopped from being executed. The present application is not limited thereto and will not be exemplified one by one here.

[0103] Figure 6 is a schematic diagram of the subject tracking method according to the embodiment of the present application. As Figure 6 shown, the method includes:

[0104] 601. Obtain the image data of the Nth frame to the N+Mth frame including the subject captured by the imaging device;

[0105] 602. Detect the information of the anatomical key points in each frame of the multi-frame image data;

[0106] 603. Select the anatomical key points with a confidence level higher than the threshold from the detected anatomical key points;

[0107] 604. Determine the reference position of the subject in each frame of the image data according to the information of the selected anatomical key points;

[0108] 605. Determine the status of the subject based on at least one of the reference position and anatomical key points in the multi-frame image data, set N to be equal to N + M + 1, and return to 601.

[0109] The implementation manners of 601 - 605 are as described above and will not be elaborated here.

[0110] Figure 7 It is a schematic diagram of the subject medical imaging method according to an embodiment of the present application. As Figure 7 shown, for example, after the subject enters the scanning room, the method includes:

[0111] 701. Obtain multi-frame image data including the subject captured by the imaging device;

[0112] 702. Detect the information of the anatomical key points in each frame of the multi-frame image data;

[0113] 703. Select the anatomical key points with a confidence level higher than the threshold from the detected anatomical key points;

[0114] 704. Determine the reference position of the subject in each frame of the image data according to the information of the selected anatomical key points;

[0115] 705. Determine that the moving direction of the subject is towards the bed board according to the reference position in the multi-frame image data;

[0116] 706. Trigger the start of explicitly displaying the detected anatomical key points of the subject on the user interface of the medical imaging system display;

[0117] 707. Determine that the subject is stably located within the bed board according to the reference position in the multi-frame image data; and perform the determination of 709 and 711.

[0118] 708. Trigger automatic positioning and subject direction recognition; optionally, the positioning and direction recognition results can also be displayed on the user interface;

[0119] 709. Determine that the status of the subject is partially occluded according to the change in the number and position of the anatomical key points in the multi-frame image data;

[0120] 710. Trigger the execution of the occlusion algorithm to detect the blanket and coil (position, etc.);

[0121] 711. Determine that the moving direction of the subject is away from the bed board according to the reference position in the multi-frame image data;

[0122] 712. Trigger the medical imaging system display to no longer display the detected anatomical key points of the subject on the user interface;

[0123] In addition, during the process from 705 to 712, the posture of the subject can also be determined as getting up, lying down, or falling according to at least one of the following changes: the change in the position of anatomical key points in the multi-frame image data, the change in the distance between anatomical key points, and the change in the depth of anatomical key points. When it is determined that the subject has fallen, the medical imaging system can also be triggered to issue an alarm, which will not be exemplified one by one here.

[0124] It should be noted that the multi-frame images in 705, 707, 709, and 711 can be multi-frame images with different time sequences or the same multi-frame images. For example, in 705, the subject's state is determined according to the Nth to (N + M1)th frame images; in 707, the subject's state is determined according to the (N + M1 + 1)th to (N + M1 + M2)th frame images; in 709, the subject's state is determined according to the (N + M1 + M2 + 1)th to (N + M1 + M2 + M3)th frame images; in 711, the subject's state is determined according to the (N + M4)th to (N + M4 + M5)th frame images, where M1, M2, M3, M4, and M5 are integers greater than 1, and M1, M2, M3, and M5 can be the same or different. The embodiments of the present application are not limited thereto.

[0125] It should be noted that the above Appendices Figure 2 , 6, and 7 only illustrate the embodiments of the present application schematically, but the present application is not limited thereto. For example, the execution order between various operations can be adjusted appropriately. In addition, some other operations can be added or some of the operations can be reduced. Those skilled in the art can make appropriate modifications according to the above content, not limited to the records in the above Appendices Figure 2 , 6, and 7.

[0126] Through the above embodiments, the state of the subject is determined according to the anatomical key points identified in the images captured by the imaging device, so as to realize automatic tracking after the subject enters the scanning room, and the determined state of the subject can be used as a trigger condition for other procedures during the scanning process, such as automatic positioning, automatic direction recognition, user interface display, execution of occlusion algorithms, etc., to further automate the scanning process and improve the scanning efficiency.

[0127] In addition, when it is detected that the subject gets up, leaves the bed board, or falls, the currently running program can be stopped in time. For example, the programs of automatic positioning, automatic direction recognition, user interface display, and occlusion algorithm can be stopped to avoid incorrect execution of the program. In addition, an alarm can also be issued to remind the clinical operator.

[0128] In addition, even when both the subject and the operator are inside the scanning room, the real-time tracking of the subject's state can also assist in the processing of positioning or direction recognition, providing more reference information for the above processing. For example, triggering the display of more information on the user interface.

[0129] An embodiment of the present application further provides a medical imaging system. Figure 3 It is a schematic diagram of the medical imaging system according to the embodiment of the present application. As Figure 3 shown, the system 300 includes:

[0130] An imaging device 301 that captures multiple frames of image data including a subject.

[0131] A controller 302, which is connected to the imaging device 301 and is configured to: obtain multiple frames of image data including a subject captured by the imaging device; detect information of anatomical key points in each frame of the multiple frames of image data; and determine the state of the subject based on the information of the anatomical key points.

[0132] Embodiments of the imaging device 301 and the controller 302 can refer to the foregoing embodiments and will not be repeated here. The medical imaging system includes but is not limited to a computed tomography (CT) system, a magnetic resonance imaging (MRI) system, a C-arm imaging system, a positron emission tomography (PET) system, a single photon emission computed tomography (SPECT) system, an ultrasound system, an X-ray imaging system, or any other suitable medical imaging system.

[0133] In some embodiments, the controller 302 can be separately configured from the controller of the medical imaging system itself. For example, the controller 302 can be configured as a chip connected to the controller of the medical imaging system, and the two can control each other. Alternatively, the functions of the controller 302 can also be integrated into the controller of the medical imaging system itself. The embodiments of the present application do not limit this.

[0134] In some embodiments, the controller 302 includes a computer processor and a storage medium, and a program for predetermined data processing to be executed by the computer processor is recorded on the storage medium. For example, a program for implementing subject tracking for medical imaging can be stored on the storage medium. The above storage medium can include, for example, ROM, floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, or non-volatile memory card.

[0135] The medical imaging system may further include structural components not shown in other figures. Specifically, reference can be made to related technologies. The embodiments of the present application do not limit this. Taking the magnetic resonance imaging system as an example, Figure 4 It is a schematic diagram of the magnetic resonance imaging system according to the embodiment of the present application. AsFigure 4 As shown, different from Figure 1 , this magnetic resonance imaging system further includes a camera device 41 that captures multiple frames of image data including the subject. Additionally, its controller 130 or computer system 120 is connected to the camera device 41 and is configured to: obtain multiple frames of image data including the subject captured by the camera device; detect information on anatomical key points in each frame of the multiple frames of image data; determine the status of the subject based on the information on the anatomical key points. Optionally, after determining the status of the subject, trigger at least one of a display change of the user interface of the medical imaging system, or trigger an automatic positioning process, or trigger a process of automatically identifying the direction of the subject, or trigger a blanket and coil shielding algorithm, or issue an alarm. The specific implementation manner can refer to the foregoing embodiments and will not be repeated here.

[0136] An embodiment of the present application further provides a subject tracking device for medical imaging. Figure 5 is a schematic diagram of the subject tracking device for medical imaging according to an embodiment of the present application. As Figure 5 shown, the device 500 includes:

[0137] An acquisition unit 501 that acquires multiple frames of image data including the subject captured by the camera device;

[0138] A detection unit 502 that detects information on anatomical key points in each frame of the multiple frames of image data;

[0139] A determination unit 503 that determines the status of the subject based on the information on the anatomical key points.

[0140] The implementation manners of the above-mentioned respective unit modules can refer to 201 to 203, and the repeated parts will not be elaborated.

[0141] Optionally, the device may further include: a control unit 504 that, after the determination unit 503 determines the status of the subject, controls at least one of a display change of the user interface of the medical imaging system, or controls the execution of an automatic positioning process, or automatically identifies the direction of the subject, or controls the execution of a blanket and coil shielding algorithm, or controls the issuance of an alarm.

[0142] An embodiment of the present application further provides a computer-readable program, wherein when the program is executed in a device or a medical imaging system, the program causes the computer to execute the subject tracking method for medical imaging described in the foregoing embodiments in the device or the medical imaging system.

[0143] An embodiment of the present application further provides a storage medium storing a computer-readable program, wherein the computer-readable program causes the computer to execute the subject tracking method for medical imaging described in the foregoing embodiments in a device or a medical imaging system.

[0144] An embodiment of the present application further provides a computer program product, at least including a computer program / instructions, and when the computer program / instructions are executed by a processor, they execute the subject tracking method for medical imaging described in the foregoing embodiments.

[0145] The above devices and methods of the present application can be implemented by hardware or by a combination of hardware and software. The present application relates to such a computer-readable program that, when executed by a logic component, can enable the logic component to implement the devices or components described above, or enable the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0146] The method / device described in combination with the embodiments of the present application can be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or a combination of one or more of the functional block diagrams can correspond to the respective software modules of the computer program flow, and can also correspond to the respective hardware modules. These software modules can respectively correspond to the respective steps shown in the figure. These hardware modules can be implemented by solidifying these software modules using a field-programmable gate array (FPGA).

[0147] The software module can be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium can be a component of the processor. The processor and the storage medium can be located in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card insertable into the mobile terminal. For example, if the device (such as a mobile terminal) uses a larger-capacity MEGA-SIM card or a large-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0148] One or more of the functional blocks described in the accompanying drawings and / or one or more combinations of functional blocks can be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in the present application. One or more of the functional blocks described in the accompanying drawings and / or one or more combinations of functional blocks can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication combination with a DSP, or any other such configuration.

[0149] The present application has been described in conjunction with specific embodiments, but those skilled in the art should understand that these descriptions are exemplary and not a limitation on the scope of the present application. Those skilled in the art can make various variations and modifications to the present application based on the principles of the present application, and these variations and modifications are also within the scope of the present application.

Claims

1. A method for tracking a subject for medical imaging, characterized in that, The method includes: Obtaining multiple frames of image data including a subject captured by an imaging device; Detecting information of anatomical key points in each frame of the multiple frames of image data; Determining the state of the subject according to the information of the anatomical key points.

2. The method according to claim 1, wherein The image data includes at least one of two-dimensional optical image data and depth image data.

3. The method according to claim 1, wherein, The information of the anatomical key points includes at least one of the following information: the anatomical type of the key point, the position coordinates of the key point, the confidence of the key point, and the depth information of the key point.

4. The method according to claim 1, wherein, The method further includes: Selecting anatomical key points with a confidence higher than a threshold from the detected anatomical key points; And determining the state of the subject according to the information of the selected anatomical key points.

5. The method according to claim 1 or 4, wherein Determining the state of the subject according to the information of the anatomical key points includes: Determining a reference position of the subject in each frame of image data according to the information of the anatomical key points; Determining whether the state of the subject is stationary or in motion according to the reference positions in the multiple frames of image data.

6. The method according to claim 5, wherein The method further includes: Determining the moving direction of the subject or the position when stationary according to at least one of the comparison result between the reference position in the multiple frames of image data and a reference position, or the depth information of the reference position in the multiple frames of image data.

7. The method according to claim 1 or 4, wherein, Determining the state of the subject according to the information of the anatomical key points includes: Determining whether the subject is partially occluded according to the change in the number and position of the anatomical key points in the multiple frames of image data.

8. The method according to claim 1 or 4, wherein Determining the state of the subject according to the information of the anatomical key points includes: Determining the posture change of the subject according to at least one of the position change of the anatomical key points in the multiple frames of image data, the distance change between the anatomical key points, and the depth change of the anatomical key points.

9. The method according to claim 1, wherein The anatomical key points include at least two of the top of the head, shoulders, nose, eyes, ears, arms, elbows, wrists, hips, knees, heart, pelvis, abdomen, chest, and ankles.

10. The method according to claim 1, wherein, The method further includes: After determining the state of the subject, triggering at least one of a display change of the user interface of the medical imaging system, or triggering an automatic positioning process, or triggering a process of automatically identifying the direction of the subject, or triggering a blanket and coil shielding algorithm, or issuing an alarm.

11. A computer-readable storage medium, the computer-readable storage medium comprising a stored computer program, wherein, When the computer program is run, it executes the method for tracking a subject for medical imaging according to any one of claims 1 to 10.

12. A medical imaging system, characterized in that, The system includes: An imaging device that captures multiple frames of image data including a subject; A controller connected to the imaging device for executing the method for tracking a subject for medical imaging according to any one of claims 1-10.

Images