Method and device for determining orientation of object in medical imaging and medical imaging system

Abstract

The embodiment of the invention provides a method and device for determining the orientation of an object in medical imaging and a medical imaging system, and the method comprises the steps: obtaining an image sequence of the object shot by a camera, and the image sequence comprises a plurality of images based on a time sequence; judging whether the state of the object is abnormal or not based on the image sequence; and determining an orientation of the object by using a first orientation or a second orientation in response to a determination result that the state of the object is abnormal, the first orientation being an orientation of the object determined based on an image prior to a period in which the state of the object is determined to be abnormal, and the second orientation being an orientation of the object determined based on an image prior to a period in which the state of the object is determined to be abnormal. The second orientation is an orientation of the object determined based on the image after a period in which the state of the object is determined to be abnormal. According to the invention, the orientation of the object can be determined more accurately.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of medical equipment technology, and in particular, to a method, device and medical imaging system for determining the orientation of an object in medical imaging. Background Art

[0002] When a medical imaging system is used to scan and image an object to be detected, it is necessary to set the orientation of the object to be detected. When the actual orientation of the object to be detected is the same as the set orientation, the medical imaging system can perform correct scanning and imaging processing on the object to be detected.

[0003] The orientation of the detected object may include which part of the body enters the scanning space first, for example, head-in means that the head of the detected object enters the scanning space first, feet-in means that the feet of the detected object enter the scanning space first, etc. The orientation of the detected object may also include which direction the face of the detected object is facing, for example, whether the detected object is prone or supine, etc.

[0004] In a medical imaging system with a camera, an image of the detected object may be captured by the camera, and the orientation of the detected object may be determined based on the image.

[0005] It should be noted that the above introduction to the technical background is only intended to provide a clear and complete description of the technical solution of the present application and to facilitate understanding by those skilled in the art. Summary of the invention

[0006] The inventor of the present application has found that when judging the orientation of the detected object based on the current image of the detected object, if the state of the detected object is abnormal, the accuracy of the orientation judgment result will be reduced. For example, when the detected object is blocked by foreign objects such as blankets, mats or coils, the camera cannot capture the image of the blocked part of the detected object, making it difficult to accurately extract key points, and thus it is impossible to accurately judge the orientation; for another example, when the detected object is in motion, the orientation cannot be accurately judged due to the possibility of orientation change.

[0007] In order to solve the above technical problems or at least similar technical problems, the embodiments of the present application provide a method, device and medical imaging system for determining the orientation of an object in medical imaging. The method determines the orientation of an object by referring to information in the time dimension, and thus can more accurately determine the orientation of the object.

[0008] According to one aspect of an embodiment of the present application, a method for determining an orientation of an object in medical imaging is provided, the method comprising:

[0009] Acquire an image sequence of an object captured by a camera, wherein the image sequence includes a plurality of images based on a time sequence;

[0010] Based on the image sequence, determining whether the state of the object is abnormal; and

[0011] In response to a determination result that the state of the object is abnormal, determining the orientation of the object using the first orientation or the second orientation,

[0012] The first orientation is the orientation of the object determined based on an image before the state of the object is determined to be abnormal, and the second orientation is the orientation of the object determined based on an image after the state of the object is determined to be abnormal.

[0013] According to one aspect of an embodiment of the present application, a device for determining an orientation of an object in medical imaging is provided, the device comprising:

[0014] an acquisition unit, which acquires an image sequence of the object captured by a camera, wherein the image sequence includes a plurality of images based on a time sequence;

[0015] a judgment unit, which judges whether the state of the object is abnormal based on the image sequence; and

[0016] an orientation determining unit, which determines the orientation of the object using the first orientation or the second orientation in response to a judgment result that the state of the object is abnormal,

[0017] The first orientation is the orientation of the object determined based on an image before the state of the object is determined to be abnormal, and the second orientation is the orientation of the object determined based on an image after the state of the object is determined to be abnormal.

[0018] According to one aspect of an embodiment of the present application, a medical imaging system is provided, the system comprising: a controller configured to execute the above method for determining an orientation of an object.

[0019] One of the beneficial effects of the embodiments of the present application is that the orientation of the object is determined based on the orientation before or after the state of the object is judged to be abnormal. Since the orientation of the object is determined using information in the time dimension, the accuracy is improved.

[0020] With reference to the following description and accompanying drawings, the specific implementation of the embodiment of the present application is disclosed in detail, indicating the way in which the principle of the embodiment of the present application can be adopted. It should be understood that the implementation of the present application is not limited in scope. Within the scope of the spirit and clauses of the attached claims, the implementation of the present application includes many changes, modifications and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 is a schematic diagram of a magnetic resonance imaging system according to an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a method for determining an orientation of an object according to an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the first orientation;

[0025] Figure 4 is a schematic diagram of the second orientation;

[0026] Figure 5 is a schematic diagram of operation 203;

[0027] Figure 6 is a flowchart of an example of determining the orientation of an object in an embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a device for determining an orientation of an object according to an embodiment of the present application;

[0029] Figure 8 is a schematic diagram of a medical imaging system. DETAILED DESCRIPTION

[0030] With reference to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations and equivalents that fall within the scope of the attached claims.

[0031] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or temporal order of these elements, etc., and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0032] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0033] In the embodiments of the present application, the term "key point" can be equivalently replaced by "key coordinate point", "landmark" or "landmark point", etc.; the term "object" can be equivalently replaced by "detection object", "detected object", "scanned object", "object to be scanned", "patient" or "research object", etc., which can be a person, an animal or other object, etc.

[0034] In the embodiments of the present application, the term "include / comprises" when used herein refers to the existence of features, whole parts, steps or components, but does not exclude the existence or addition of one or more other features, whole parts, steps or components.

[0035] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0036] In an embodiment of the present application, the method for determining the orientation of an object or the device for determining the orientation of an object can be applicable to various medical imaging scenarios, including but not limited to magnetic resonance imaging (MRI), computed tomography (CT), ultrasound imaging (Ultrasound Imaging), positron emission tomography (PET), single photon emission computed tomography (SPECT), PET / CT, PET / MR or any other suitable medical imaging scenario.

[0037] In the embodiments of the present application, the method, device and system of the present application are exemplarily described by taking the MRI scenario as an example. It can be understood that the contents of the embodiments of the present application are also applicable to other medical imaging scenarios.

[0038] For ease of understanding, Figure 1 FIG. 1 is a schematic diagram of a magnetic resonance imaging (MRI) system 100 according to an embodiment of the present application.

[0039] The MRI system 100 includes a scanning unit 111. The scanning unit 111 is used to perform magnetic resonance scanning on an object (eg, a human body) 170 to generate image data of a region of interest of the object 170, where the region of interest may be a predetermined anatomical site or anatomical tissue.

[0040] The operation of the MRI system 100 is controlled by an operator workstation 110, which includes an input device 114, a control panel 116, and a display 118. The input device 114 can be a joystick, a keyboard, a mouse, a trackball, a touch-activated screen, voice control, or any similar or equivalent input device. The control panel 116 can include a keyboard, a touch-activated screen, voice control, buttons, a slider, or any similar or equivalent control device. The operator workstation 110 is coupled to and communicates with a computer system 120, which enables an operator to control the generation and viewing of images on the display 118. The computer system 120 includes multiple components that communicate with each other via electrical and / or data connection modules 122. The connection module 122 can be a direct wired connection, an optical fiber connection, a wireless communication link, etc. The computer system 120 may include a central processing unit (CPU) 124, a memory 126, and an image processor 128. In some embodiments, the image processor 128 can be replaced by a medical imaging function implemented in the CPU 124. The computer system 120 may be connected to an archival media device, permanent or backup storage, or a network. The computer system 120 may be coupled to and communicate with a separate MRI system controller 130 .

[0041] The MRI system controller 130 includes a set of components that communicate with each other via an electrical and / or data connection module 132. The connection module 132 can be a direct wired connection, an optical fiber connection, a wireless communication link, etc. The MRI system controller 130 may include a CPU 131, a sequence pulse generator (also referred to as a pulse generator) 133 that communicates with the operator workstation 110, a transceiver (also referred to as an RF transceiver) 135, a memory 137, and an array processor 139.

[0042] In some embodiments, the sequence pulse generator 133 can be integrated into the resonance assembly 140 of the scanning unit 111 of the MRI system 100. The MRI system controller 130 can receive commands from the operator workstation 110, coupled to the scanning unit 111, to indicate the MRI scanning sequence to be performed during the MRI scan, so as to control the scanning unit 111 to perform the above-mentioned magnetic resonance scanning process. The MRI system controller 130 is also coupled to and communicates with the gradient drive system (also referred to as gradient drive) 150, which is coupled to the gradient coil assembly 142 to generate a magnetic field gradient during the MRI scan.

[0043] The sequence pulse generator 133 may also receive data from a physiological acquisition controller 155, which receives signals from a plurality of different sensors (e.g., electrocardiogram (ECG) signals from electrodes attached to the patient, etc.) connected to a subject or patient 170 undergoing an MRI scan. The sequence pulse generator 133 is coupled to and in communication with a scan room interface system 145, which receives signals from various sensors associated with the state of the resonance assembly 140. The scan room interface system 145 is also coupled to and in communication with a patient positioning system 147, which sends and receives signals to control the movement of the patient table to a desired position for an MRI scan.

[0044] The MRI system controller 130 provides gradient waveforms to the gradient driver system 150, which includes a G x (x direction), G y (y direction) and G z (z direction) amplifier, etc. Each G x , G y and G z The gradient amplifiers all excite corresponding gradient coils in the gradient coil assembly 142 to generate magnetic field gradients for spatially encoding MR signals during MRI scanning. The gradient coil assembly 142 is disposed within a resonance assembly 140, which also includes a superconducting magnet having superconducting coils 144, which in operation provide a static uniform longitudinal magnetic field B0 throughout a cylindrical imaging volume 146. The resonance assembly 140 also includes an RF body coil 148, which in operation provides a transverse magnetic field B1 that is substantially perpendicular to B0 throughout the cylindrical imaging volume 146. The resonance assembly 140 may also include an RF surface coil 149 for imaging different anatomical structures of a patient undergoing an MRI scan. The RF body coil 148 and the RF surface coil 149 may be configured to operate in a transmit and receive mode, a transmit mode, or a receive mode.

[0045] The x direction can also be called the frequency encoding direction or the k direction in K space. x The y direction can be called the phase encoding direction or the k direction in K space. y Direction, the z direction can be called the layer selection direction. x It can be used for frequency encoding or signal readout and is usually called frequency encoding gradient or readout gradient. y Can be used for phase encoding, usually called phase encoding gradient. G z It can be used for slice (layer) position selection to obtain K-space data. It should be noted that the slice selection direction, phase encoding direction and frequency encoding direction can be modified according to actual needs.

[0046] An MRI scanned subject or patient 170 may be positioned within a cylindrical imaging volume 146 of the resonance assembly 140. The transceiver 135 in the MRI system controller 130 generates RF excitation pulses that are amplified by an RF amplifier 162 and provided to the RF body coil 148 via a transmit / receive switch (also referred to as a T / R switch or switch) 164.

[0047] As described above, the RF body coil 148 and the RF surface coil 149 can be used to transmit RF excitation pulses and / or receive the resulting MR signals from the patient undergoing the MRI scan. The MR signals emitted by the nuclei excited in the patient's body of the MRI scan can be sensed and received by the RF body coil 148 or the RF surface coil 149 and sent back to the preamplifier 166 through the T / R switch 164. The T / R switch 164 can be controlled by a signal from the sequence pulse generator 133 to electrically connect the RF amplifier 162 to the RF body coil 148 during the transmit mode, and to connect the preamplifier 166 to the RF body coil 148 during the receive mode. The T / R switch 164 can also enable the RF surface coil 149 to be used in the transmit mode or the receive mode.

[0048] In some embodiments, the MR signals sensed and received by the RF body coil 148 or the RF surface coil 149 and amplified by the preamplifier 166 are stored as a raw k-space data array in the memory 137 for post-processing. A reconstructed magnetic resonance image can be acquired by transforming / processing the stored raw k-space data.

[0049] In some embodiments, the MR signals sensed and received by the RF body coil 148 or the RF surface coil 149 and amplified by the preamplifier 166 are demodulated, filtered, and digitized in the receive portion of the transceiver 135 and transmitted to the memory 137 in the MRI system controller 130. For each image to be reconstructed, the data is rearranged into a separate k-space data array, and each of these separate k-space data arrays is input to an array processor 139, which is operated to Fourier transform the data into an array of image data.

[0050] The array processor 139 uses a transform method, most commonly a Fourier transform, to create images from the received MR signals. These images are transmitted to the computer system 120 and stored in the memory 126. In response to commands received from the operator workstation 110, the image data may be stored in long-term storage or may be further processed by the image processor 128 and transmitted to the operator workstation 110 for presentation on the display 118.

[0051] In various embodiments, components of computer system 120 and MRI system controller 130 may be implemented on the same computer system or multiple computer systems. Figure 1 The MRI system 100 shown is for illustration purposes. Suitable MRI systems may include more, fewer, and / or different components.

[0052] The MRI system controller 130 and the image processor 128 may include a computer processor and a storage medium, respectively or in common, on which a program for predetermined data processing to be executed by the computer processor is recorded, for example, the storage medium may store a program for implementing scanning processing (e.g., scanning process, imaging sequence), image reconstruction, medical imaging, etc. For example, a computer program for determining the orientation of an object for implementing an embodiment of the present invention may be stored. The above-mentioned storage medium may include, for example, a ROM, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, or a non-volatile memory card.

[0053] The inventors have found that when using images captured by a camera to determine the orientation of an object, in some medical scenarios, it is necessary to cover the surface of the object with an obstruction (e.g., a coil, a mat, or a blanket, etc.). For example, in the MRI scenario described above, it is necessary to cover the surface of the object 170 with an RF surface coil 149, etc. Since part of the object is covered by the obstruction, the camera cannot obtain the image within the coverage area, making it difficult to accurately extract key points, and thus unable to accurately determine the orientation. In addition, when the object being detected is in motion, it is also difficult to accurately determine the orientation due to the possibility of orientation change.

[0054] In response to at least one of the above problems, embodiments of the present application provide a method, an apparatus, and a medical imaging system for determining an orientation of an object in medical imaging.

[0055] The following is a description with reference to embodiments.

[0056] The present application embodiment provides a method for determining the orientation of an object in medical imaging. Figure 2 is a schematic diagram of a method for determining an orientation of an object in medical imaging according to an embodiment of the present application. Figure 2 As shown, the method includes:

[0057] Operation 201: acquiring an image sequence of an object captured by a camera, where the image sequence includes a plurality of images based on a time sequence;

[0058] Operation 202: judging whether the state of the object is abnormal based on the image sequence; and

[0059] Operation 203: In response to a judgment result that the state of the object is abnormal, determine the orientation of the object using the first orientation or the second orientation.

[0060] In the present application, the first orientation is the orientation of the object determined based on an image before the state of the object is determined to be abnormal, and the second orientation is the orientation of the object determined based on an image after the state of the object is determined to be abnormal.

[0061] Figure 3 This is a schematic diagram of the first orientation. Figure 4 is a schematic diagram of the second orientation. Figure 3 As shown, from time T1, the state of the object is judged to be abnormal, and the abnormal state can continue, so the image before the state of the object is judged to be abnormal can refer to the image before time T1, and the object orientation determined based on the image before time T1 is the first orientation. Figure 4 As shown, from moment T1, the state of the object is judged to be abnormal, and at moment T2, the state of the object is judged to recover from abnormal to normal, that is, the period during which the state of the object is judged to be abnormal refers to the period from moment T1 to moment T2, so the image after the period during which the state of the object is judged to be abnormal can refer to the image after moment T2, and the object orientation determined based on the image after moment T2 is the second orientation.

[0062] According to the above embodiment, the orientation of the object is determined based on the orientation before or after the period in which the state of the object is determined to be abnormal. Since the orientation of the object is determined using information in the time dimension, accuracy is improved.

[0063] In some embodiments, the orientation of the detected object may include which part of the body enters the scanning space first, for example, head in means that the head of the detected object enters the scanning space first, feet in means that the feet of the detected object enter the scanning space first, etc. The orientation of the detected object may also include which direction the face of the detected object is facing, for example, whether the detected object is prone or supine, etc.

[0064] In some embodiments, the image sequence may include multiple images, and the multiple images form an image sequence in chronological order. Each image in the image sequence may include a color image and a depth image.

[0065] Each color image may include a plurality of pixels, and the value of each pixel may include the value of each color component, wherein the color component may include a red (R) component, a green (G) component, a blue (B) component, and the like.

[0066] Each depth image may include a plurality of pixels, and the value of each pixel may include a depth value of the pixel, wherein the depth value of the pixel may represent the distance from the point on the object corresponding to the pixel to the camera.

[0067] In some embodiments, the camera may acquire a depth image sequence and a color image sequence over a period of time. The color image sequence may include multiple color images, and the depth image sequence may include multiple depth images, each color image corresponds to a different moment, and each depth image corresponds to a different moment. Among them, each color image in the color image sequence may correspond one-to-one to each depth image in the depth image sequence, for example, each color image in the color image sequence may be aligned in time with each depth image in the depth image sequence. The present application is not limited to this, and each color image in the color image sequence may have other corresponding relationships with each depth image in the depth image sequence.

[0068] In some embodiments, the camera may be placed Figure 1 The scanning unit 111 is positioned near the scanning unit 111 (for example, on the upper side) so as to capture the object 170 from a bird's-eye view to obtain an image. Figure 1 The position of the camera 180 is shown. In addition, the present application is not limited thereto, and the camera may also be set at other positions.

[0069] In the present application, the state of an object is abnormal, including: the object is blocked, and / or the object is moving. The object is blocked, for example, it means that the object is blocked by an object such as a coil or a blanket. The object is moving, which means that at least a part of the object's body is in motion, for example, the object changes from lying on its back to lying on its stomach, or the object stands up from lying on its back to sitting, etc. In addition, the state of an object is abnormal, and this application is not limited to the object being blocked or the object being in motion, etc.

[0070] In operation 202 , it may be determined whether the state of an object in an image is abnormal based on a depth image and / or a color image in an image sequence.

[0071] In some examples, it is possible to determine whether an object is occluded based on the depth image and the color image. For example, the following operations are used to determine whether an object is occluded:

[0072] Operation 1: Input a color image into a neural network, and the neural network outputs the confidence of each key point of the object in the color image, wherein the confidence can indicate the possibility that the object is occluded, for example, the lower the confidence, the higher the possibility of being occluded;

[0073] Operation 2: Use the depth image to obtain the depth value of each key point. When the object is blocked, the distance between the blocker and the camera is closer, so the depth value is smaller.

[0074] Operation 3: when the confidence of the key point is lower than the first threshold and the depth value of the key point is less than the second threshold, the key point is determined to be occluded; and when the number of key points determined to be occluded reaches a third threshold, the object is determined to be occluded.

[0075] In the present application, the key point may be, for example, a feature point related to a body part that can be detected in an image. The position information of the key point may be represented by a pixel position in the image. The present application is not limited thereto, and the position information of the key point may also be represented by other means.

[0076] Among them, the type of key point may include at least one of the following: head, chest, abdomen, neck, nose, left shoulder, right shoulder, left hip, right hip, left eye, right eye, left elbow, right elbow, left knee, right knee, left ear, right ear, left wrist, right wrist, left ankle and right ankle. The present application is not limited to this, and the key point type may also include other content.

[0077] The confidence of the key point may be in the form of probability, for example, the probability that the real position of the key point is located at the detected position, etc. The present application is not limited thereto, and the confidence of the key point may also be in other forms or have other meanings.

[0078] In other examples, whether the state of the object is blocked can be determined based on the depth image. For example, the depth value of each key point is obtained using the depth image, and when the depth value of the key point is less than the second threshold, the key point is determined to be blocked, and when the number of key points determined to be blocked reaches the third threshold, the object is determined to be blocked.

[0079] In some other examples, it is possible to determine whether the state of an object is obscured based on a color image. For example, a color image is input into a neural network, and the neural network outputs the confidence of each key point of the object in the color image. If the confidence of the key point is lower than a first threshold, the key point is determined to be obscured, and when the number of key points determined to be obscured reaches a third threshold, the object is determined to be obscured. The neural network can be trained, for example, in the following manner: an image set of known key points and key point confidences is used as an input set, and the known key point confidences are used as an output set to train the neural network.

[0080] In operation 202 of the present application, whether the object is blocked can be determined based on the blocking state of the key points of the object and the time information of the blocking state. For example, Table 1 is an example of the blocking state and the time information of the blocking state.

[0081] Table 1

[0082] Moment 1 … Time p … Time M1 Key Point 1 1 … 0 … 1 … … … … … Key Points 1 … 1 … 0 … … … … … Key Point M2 1 … 0 … 0

[0083] As shown in Table 1, M1 is the number of color images or depth images, and the pth image corresponds to the moment p in the time series; M2 is the total number of key points that the neural network model can detect. 0 in the table indicates that the key point is blocked at a certain moment p, and 1 in the table indicates that the key point is not blocked at a certain moment p.

[0084] In the time series, if the first "0" appears at time T1 or a number of "0"s greater than or equal to a first predetermined number appear simultaneously, it is judged that the object is obscured from time T1; after T1, if the last "0" appears at time T2 or the number of "0"s that appear simultaneously is less than or equal to a second predetermined number, it is judged that the obscured state of the object ends at time T2.

[0085] In some examples, the method of determining whether an object is moving based on a depth image and / or a color image may be, for example: determining whether the depth value of a key point has changed based on the depth image, and determining whether the position of the key point in the image has changed based on the color image. When at least one of the aforementioned changes exceeds a corresponding threshold, the object is determined to be moving.

[0086] For example, based on the depth image, it is judged that the change in the depth values ​​of the key points corresponding to the left shoulder, right shoulder, left ear and right ear exceeds the first threshold, and based on the color image, it is judged that the change in the positions of the key points corresponding to the left shoulder, right shoulder, left ear and right ear in the image exceeds the second threshold, thereby determining that the object is moving.

[0087] The above examples are only examples, and the present application is not limited thereto. Other methods may also be used to determine whether an object is abnormal.

[0088] Figure 5 is a schematic diagram of operation 203. Figure 5 As shown, in operation 203, in response to a judgment result that the state of the object is abnormal (eg, the judgment result obtained in operation 202), the orientation of the object is determined using the first orientation or the second orientation.

[0089] like Figure 5 As shown in operation 2031, in at least some embodiments, in response to the determination result that the object is blocked, the orientation of the object is determined using the first orientation. Determining the orientation of the object using the first orientation may include the following methods:

[0090] Method 1: using a first orientation corresponding to a first predetermined image before the state of the object is determined to be blocked as the orientation of the object; or

[0091] Method 2: Processing the first orientations corresponding to at least two images before the state of the object is determined to be occluded, to obtain the orientation of the object.

[0092] For method 1, for example, from time T1, it is determined that the object is blocked, so the first orientation corresponding to the first predetermined image before time T1 is used as the orientation of the object. The first predetermined image may be the image closest to time T1 in time, or an image with a preset time interval from time T1.

[0093] For method 2, for example, from time T1, it is determined that the object is blocked, so the first orientations corresponding to at least two images before time T1 are processed to obtain the orientation of the object. The number of the at least two images can be a preset number, for example, N images before time T1. The processing includes at least one of the following processing:

[0094] Processing 1: Performing weighted sum processing on the first orientations corresponding to the at least two images, for example, N images correspond to N first orientations, and different types of first orientations correspond to different values ​​(for example, the value of the head-in supine position is 1, the value of the head-in prone position is 2, the value of the feet-in supine position is 3, ..., etc.), and the closer to the time T1 in time, the greater or smaller the weight assigned to the first orientation corresponding to the image, thereby, the orientation to which the value obtained by performing the weighted sum processing on the N first orientations is closer is determined as the orientation of the object;

[0095] Process 2: averaging the first orientations corresponding to the at least two images. For example, different types of first orientations correspond to different values ​​(for example, the value of the head-in supine position is 1, the value of the head-in prone position is 2, the value of the feet-in supine position is 3, etc.), averaging the N first orientations. The orientation to which the obtained value is closer is determined as the orientation of the object.

[0096] Process 3: Select the first orientation with the largest number of occurrences. For example, if the head-forward supine position appears 4 times among N (for example, N=5) first orientations, then the head-forward supine position is determined as the orientation of the object.

[0097] Processing 4: Select the processing of the first orientation whose occurrence number exceeds the threshold. For example, the threshold is 2, and among N (for example, N=5) first orientations, the feet-in-prone position appears 3 times (greater than the threshold), the head-in-supine position appears 1 time, and the head-in-prone position appears 1 time, then the head-in-supine orientation is determined as the orientation of the object. In addition, if the occurrence number of more than two orientations exceeds the threshold, at least one of processing 3, processing 2 or processing 1 can be further used to determine the orientation of the object.

[0098] like Figure 5As shown in operation 2032, in at least some other embodiments, in response to the determination result that the object is moving, the orientation of the object is determined using the second orientation. Wherein, determining the orientation of the object using the second orientation may include the following methods:

[0099] Mode 3: using a second orientation corresponding to a second predetermined image after the state of the object is determined to be in motion as the orientation of the object; or

[0100] Method 4: Process the second orientations corresponding to at least two images after the state of the object is determined to be in motion, to obtain the orientation of the object.

[0101] For method 3, for example, during the period from time T1 to time T2, it is determined that the object is moving, and therefore, the second orientation corresponding to the second predetermined image after time T2 is used as the orientation of the object. The second predetermined image may be the image closest to time T2 in time, or an image with a preset time interval from time T2.

[0102] For method 4, for example, during the period from time T1 to time T2, it is determined that the object is moving, and therefore, the second orientations corresponding to at least two images after time T2 are processed to obtain the orientation of the object. The number of the at least two images can be a preset number, for example, M images after time T2. The processing includes at least one of the following processing:

[0103] Process 5: Perform weighted sum processing on the second orientations corresponding to the at least two images. For example, M images correspond to M second orientations, and different types of second orientations correspond to different values ​​(for example, the value of the head-in supine position is 1, the value of the head-in prone position is 2, the value of the feet-in supine position is 3, ..., etc.), and the closer to the moment T2 in time, the greater or smaller the weight assigned to the second orientation corresponding to the image. Therefore, the orientation to which the value obtained by performing the weighted sum processing on the M second orientations is closer is determined as the orientation of the object;

[0104] Process 6: averaging the second orientations corresponding to the at least two images. For example, different types of second orientations correspond to different values ​​(for example, the value of the head-in supine position is 1, the value of the head-in prone position is 2, the value of the feet-in supine position is 3, etc.), averaging the M second orientations. The orientation to which the obtained value is closer is determined as the orientation of the object.

[0105] Process 7: Select the second orientation with the largest number of occurrences. For example, if the head-forward supine position appears 4 times among M (for example, M=5) second orientations, then the head-forward supine position is determined as the orientation of the object.

[0106] Processing 8: Select the processing of the second orientation whose occurrence number exceeds the threshold. For example, the threshold is 2, and among M (for example, M=5) second orientations, the feet-in-prone position appears 3 times (greater than the threshold), the head-in-supine position appears 1 time, and the head-in-prone position appears 1 time, then the head-in-supine orientation is determined as the orientation of the object. In addition, if the occurrence number of more than two orientations exceeds the threshold, at least one of processing 7, processing 6 or processing 5 can be further used to determine the orientation of the object.

[0107] In addition, if the object is detected to be moving during the period when the object is blocked, the orientation of the object can be determined based on method 1 or method 2. For example, from time T1, it is determined that the object is blocked, and during the period from time T11 to time T2 after time T1, the object is detected to be moving, then the first orientation corresponding to the first predetermined image before time T1 is used as the orientation of the object, or the first orientations corresponding to at least two images before time T1 are processed to obtain the orientation of the object.

[0108] In addition, if Figure 5 As shown, if there is no abnormality in the state of the object, then as shown in operation 204, the orientation of the object is determined based on the current image.

[0109] In some embodiments of the present application, the first orientation or the second orientation may be obtained based on a depth image and / or a color image of the image.

[0110] For example, a color image may be input into an orientation judgment model to obtain an orientation corresponding to the color image (the orientation is, for example, a first orientation or a second orientation). The orientation judgment model may judge the orientation of the object based on a classification network, or the orientation judgment model may perform key point detection on the color image and judge the orientation of the object based on the result of the key point detection (for example, information such as the position of the key point);

[0111] For another example, the depth information of the key points in the depth image may be compared with templates of depth information corresponding to different orientations, thereby determining the orientation of the object.

[0112] The specific details of the method for determining the first orientation or the second orientation may refer to the related art. In addition, the method for determining the first orientation or the second orientation may not be limited to the above description, and other methods may also be used.

[0113] Figure 6It is a flowchart of an example of determining the orientation of an object according to an embodiment of the present application.

[0114] like Figure 6 As shown, the image sequence includes depth images 1 to K and color images 1 to K (K is a natural number greater than or equal to 2), and depth image i and color image i correspond to the same moment i (i is a natural number greater than or equal to 1 and less than or equal to K).

[0115] exist Figure 6 In the example shown, in operation 601 , the orientation of the object is determined according to each color image i, so that the orientation i corresponding to each color image i, ie, orientation 1 to orientation K, is output.

[0116] like Figure 6 As shown, in operation 602, based on the depth image i and the color image i, it is determined whether the state of the object at time i is abnormal.

[0117] like Figure 6 As shown, in operation 603, multiple orientations i are combined to output the orientation information of the object.

[0118] For example, when the state of the object at moment i is normal, operation 603 outputs orientation i; when the state of the object at moment i is abnormal, operation 603 determines the orientation of the object using the first orientation before moment i, or operation 603 determines the orientation of the object using the second orientation after the state of the object returns to normal (i.e., after the abnormality ends).

[0119] According to the above embodiment, the orientation of the object is determined based on the orientation before or after the period in which the state of the object is determined to be abnormal. Since the orientation of the object is determined using information in the time dimension, accuracy is improved.

[0120] The embodiment of the present application also provides a device for determining the orientation of an object, and the same contents as the above-mentioned embodiment will not be repeated here.

[0121] Figure 7 is a schematic diagram of a device for determining an orientation of an object according to an embodiment of the present application. Figure 7 As shown, the apparatus 700 for determining the orientation of an object in medical imaging includes:

[0122] An acquisition unit 701, which acquires an image sequence of an object captured by a camera, wherein the image sequence includes a plurality of images based on a time sequence;

[0123] A judging unit 702, which judges whether the state of the object is abnormal based on the image sequence; and

[0124] The orientation determining unit 703 determines the orientation of the object using the first orientation or the second orientation in response to the judgment result that the state of the object is abnormal.

[0125] In some embodiments, the first orientation is the orientation of the object determined based on an image before a period during which the state of the object is determined to be abnormal, and the second orientation is the orientation of the object determined based on an image after a period during which the state of the object is determined to be abnormal.

[0126] In some embodiments, the state of the object is abnormal, including:

[0127] The object is occluded, and / or the object is moving.

[0128] In some embodiments, the orientation determination unit determines the orientation of the object using the first orientation in response to a determination result that the object is blocked.

[0129] In some embodiments, determining the orientation of the object using the first orientation includes:

[0130] using the first orientation corresponding to the first predetermined image before the state of the object is determined to be blocked as the orientation of the object; or

[0131] The first orientations corresponding to at least two images before the state of the object is determined to be blocked are processed to obtain the orientation of the object.

[0132] In some embodiments, the processing includes at least one of the following:

[0133] The first orientations corresponding to the at least two images are subjected to weighted summing processing, averaging processing, selecting processing of the first orientation with the largest number of occurrences, or selecting processing of the first orientation with the number of occurrences exceeding a threshold.

[0134] In some embodiments, the orientation determination unit determines the orientation of the object using the second orientation in response to a determination that the object is moving.

[0135] In some embodiments, determining the orientation of the object using the second orientation includes:

[0136] using the second orientation corresponding to a second predetermined image after the state of the object is determined to be in motion as the orientation of the object; or

[0137] The second orientations corresponding to at least two images after the state of the object is determined to be in motion are processed to obtain the orientation of the object.

[0138] In some embodiments, the processing includes at least one of the following:

[0139] The second orientations corresponding to the at least two images are subjected to weighted summing processing, averaging processing, selecting processing of the second orientation with the largest number of occurrences, or selecting processing of the second orientation with the number of occurrences exceeding a threshold.

[0140] In some embodiments, each image in the image sequence comprises a depth image and a color image,

[0141] Wherein, the judging unit judges whether the state of the object is abnormal based on the depth image and / or the color image.

[0142] In some embodiments, the first orientation or the second orientation is obtained based on a depth image and / or a color image of the image.

[0143] It is worth noting that only the components or modules related to the present application are described above, but the present application is not limited thereto. The device 700 for determining the orientation of an object may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.

[0144] To keep it simple, Figure 7 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned various components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0145] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0146] The present application also provides a medical imaging system. The system includes a device 700 for determining the orientation of an object in medical imaging as described in the second aspect of the embodiment, the contents of which are incorporated herein. The medical imaging device may include, for example, a computer, a server, a workstation, a laptop computer, a smart phone, etc., but the present application is not limited thereto.

[0147] Figure 8 Schematic diagram of a medical imaging device according to an embodiment of the present application. Figure 8As shown, the medical imaging device 800 may include: one or more processors (e.g., central processing unit CPU) 810 and one or more memories 820; the memory 820 is coupled to the processor 810. The memory 820 may store various data; in addition, it may store a program 821 for information processing, and execute the program 821 under the control of the processor 810.

[0148] In some embodiments, the functions of the apparatus 700 for determining the orientation of an object in medical imaging are integrated into the processor 810. The processor 810 is configured to implement the method for determining the orientation of an object in medical imaging described in the above embodiments of the present application.

[0149] In some embodiments, the device 700 for determining the orientation of an object in medical imaging is configured separately from the processor 810. For example, the device 700 for determining the orientation of an object in medical imaging can be configured as a chip connected to the processor 810, and the function of the device 700 for determining the orientation of an object in medical imaging is implemented through the control of the processor 810.

[0150] For example, the processor 810 is configured to perform the following control: acquiring an image sequence of an object captured by a camera, the image sequence including a plurality of images based on a time series; judging whether the state of the object is abnormal based on the image sequence; and determining the orientation of the object using a first orientation or a second orientation in response to a judgment result that the state of the object is abnormal, wherein the first orientation is the orientation of the object determined based on an image before a period during which the state of the object is judged to be abnormal, and the second orientation is the orientation of the object determined based on an image after a period during which the state of the object is judged to be abnormal.

[0151] In a specific example, Figure 8 The medical imaging device 800 may be Figure 1 The magnetic resonance imaging (MRI) system 100 shown in FIG. Figure 8 The memory 820 may correspond to Figure 1 At least one of the memory 137 and the memory 126, for example, the memory 820 may be independent of at least one of the memory 137 and the memory 126, or, the memory 820 may communicate with at least one of the memory 137 and the memory 126, or the memory 820 may include at least one of the memory 137 and the memory 126, and so on. Figure 8 The processor 810 may correspond to Figure 1The processor 810 may be independent of at least one of the CPU 131, the CPU 124 and the image processor 128, for example, the processor 810 may be independent of the CPU 131, the CPU 124 and the image processor 128, or the processor 810 may communicate with at least one of the CPU 131, the CPU 124 and the image processor 128, or the processor 810 may include at least one of the CPU 131, the CPU 124 and the image processor 128, etc.

[0152] In addition, if Figure 8 As shown, the medical imaging device 800 may further include: an input / output (I / O) device 830 and a display 840, etc.; wherein the functions of the above components are similar to those in the prior art and are not described in detail here.

[0153] In addition, if Figure 8 As shown, the medical imaging device 800 may also include a camera 850, which photographs an object and generates an image sequence, which can be transmitted to the processor 810, so that the processor 810 can implement the method for determining the orientation of an object in medical imaging described in the above embodiment of the present application based on the image taken by the camera 850.

[0154] It is worth noting that the medical imaging device 800 does not necessarily have to include Figure 8 In addition, the medical imaging device 800 may also include Figure 8 For components not shown in the figure, reference may be made to the related art.

[0155] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a medical imaging system, the program enables a computer to execute the method for determining the orientation of an object in medical imaging described in the aforementioned embodiment in the medical imaging system.

[0156] An embodiment of the present application further provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the method for determining the orientation of an object in medical imaging described in the aforementioned embodiment in a medical imaging system.

[0157] The above devices and methods of the present application can be implemented by hardware, or by hardware combined with software. The present application relates to such a computer-readable program, which, when executed by a logic component, enables the logic component to implement the above-mentioned devices or components, or enables the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0158] The method / device described in conjunction with the embodiments of the present application may be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to the various steps shown in the figure, respectively. These hardware modules may be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).

[0159] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if a device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0160] For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present application. For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0161] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.

Claims

1. A method for determining the orientation of an object in medical imaging, characterized in that The method comprises: Acquire an image sequence of an object captured by a camera, wherein the image sequence includes a plurality of images based on a time sequence; Based on the image sequence, determining whether the state of the object is abnormal; and In response to a determination result that the state of the object is abnormal, determining the orientation of the object using the first orientation or the second orientation, The first orientation is the orientation of the object determined based on an image before the state of the object is determined to be abnormal, and the second orientation is the orientation of the object determined based on an image after the state of the object is determined to be abnormal.

2. The method according to claim 1, characterized in that Determining whether the state of the object is abnormal includes: It is determined whether at least one of: the object is blocked and whether a motion occurs.

3. The method according to claim 2, characterized in that In response to a determination that the object is blocked, the orientation of the object is determined using the first orientation.

4. The method according to claim 3, characterized in that Determining the orientation of the object using the first orientation includes: using the first orientation corresponding to the first predetermined image before the state of the object is determined to be blocked as the orientation of the object; or The first orientations corresponding to at least two images before the state of the object is determined to be blocked are processed to obtain the orientation of the object.

5. The method according to claim 4, characterized in that The processing includes: The first orientations corresponding to the at least two images are subjected to weighted summing or averaging processing.

6. The method according to claim 4, characterized in that The processing includes: The processing of the first orientation having the largest number of appearances is selected or the processing of the first orientation having the number of appearances exceeding a threshold is selected.

7. The method according to claim 2, characterized in that In response to a determination that the object is in motion, the orientation of the object is determined using the second orientation.

8. The method according to claim 7, characterized in that Determining the orientation of the object using the second orientation includes: using the second orientation corresponding to a second predetermined image after the state of the object is determined to be in motion as the orientation of the object; or The second orientations corresponding to at least two images after the state of the object is determined to be in motion are processed to obtain the orientation of the object.

9. The method according to claim 8, characterized in that The processing includes: The second orientations corresponding to the at least two images are subjected to weighted summing or averaging processing.

10. The method according to claim 8, characterized in that The processing includes: The processing of the second orientation having the largest number of appearances is selected or the processing of the second orientation having the number of appearances exceeding a threshold is selected.

11. A medical imaging system, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method for determining an orientation of an object in medical imaging according to any one of claims 1 to 10.

12. A device for determining the orientation of an object in medical imaging, characterized in that: The device comprises: an acquisition unit, which acquires an image sequence of the object captured by a camera, wherein the image sequence includes a plurality of images based on a time sequence; a judging unit, which judges whether the state of the object is abnormal based on the image sequence; and an orientation determining unit, which determines the orientation of the object using the first orientation or the second orientation in response to a judgment result that the state of the object is abnormal, The first orientation is the orientation of the object determined based on an image before the state of the object is determined to be abnormal, and the second orientation is the orientation of the object determined based on an image after the state of the object is determined to be abnormal.

Images