Key point detection method for medical imaging and medical imaging method and system

Abstract

The embodiment of the invention provides a key point detection method for medical imaging, a medical imaging method and a medical imaging system.The key point detection method for medical imaging comprises the steps that an image sequence is received, and the image sequence comprises a plurality of images of an object; performing key point detection on the plurality of images to generate a key point image sequence; carrying out key point shielding detection on images in the image sequence; and determining key point distribution information according to the key point image sequence and a key point shielding detection result.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of medical equipment technology, and in particular to a key point detection method for medical imaging, a medical imaging method and a system. Background Art

[0002] In medical scenarios, landmark points can refer to key coordinate points that are anatomically important, usually the junctions of different tissues and organs or the identification points with the most prominent morphological features of the research object. These landmark points can be used in medicine to identify tissue structures. For example, in a magnetic resonance imaging (MRI) scenario, landmark points of the scanned object can be used to identify the tissue or organ corresponding to the landmark point, thereby determining the area to be scanned of the scanned object. Summary of the invention

[0003] Embodiments of the present application provide a key point detection method, a medical imaging method, and a system for medical imaging.

[0004] According to one aspect of an embodiment of the present application, a key point detection method for medical imaging is provided, the method comprising: receiving an image sequence, the image sequence comprising multiple images of an object; performing key point detection on the multiple images respectively to generate a key point image sequence; performing key point occlusion detection on the images in the image sequence; and determining key point distribution information based on the key point image sequence and the results of the key point occlusion detection.

[0005] According to one aspect of an embodiment of the present application, a medical imaging method is provided, the method comprising: determining key point distribution information according to the above-mentioned key point detection method for medical imaging; and performing a scanning operation according to the determined key point distribution information.

[0006] According to one aspect of an embodiment of the present application, a medical imaging system is provided, the system comprising: a controller configured to execute the above-mentioned key point detection method for medical imaging; and a scanning component that performs a scanning operation according to key point distribution information determined by the controller.

[0007] One of the beneficial effects of the embodiments of the present application is that key point detection is performed on multiple images in an image sequence respectively to generate a key point image sequence, and key point occlusion detection is performed on multiple images in the image sequence, and key point distribution information is determined based on the generated key point image sequence and the results of the key point occlusion detection, thereby improving the accuracy and reliability of the key point distribution information.

[0008] With reference to the following description and accompanying drawings, the specific implementation of the embodiment of the present application is disclosed in detail, indicating the way in which the principle of the embodiment of the present application can be adopted. It should be understood that the implementation of the present application is not limited in scope. Within the scope of the spirit and clauses of the attached claims, the implementation of the present application includes many changes, modifications and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without creative work. In the drawings:

[0010] Figure 1 is a schematic diagram of a magnetic resonance imaging system according to an embodiment of the present application;

[0011] Figure 2 is a schematic diagram of a key point detection method for medical imaging according to an embodiment of the present application;

[0012] Figure 3 is a schematic diagram of an implementation of step 202 of an embodiment of the present application;

[0013] Figure 4 It is a schematic diagram of the process of step 202 of the embodiment of the present application;

[0014] Figure 5 is a schematic diagram of the occlusion state of key points in an embodiment of the present application;

[0015] Figure 6 is a schematic diagram of an implementation of step 204 of an embodiment of the present application;

[0016] Figure 7 is a schematic diagram of an implementation of step 604 of an embodiment of the present application;

[0017] Figure 8 is a schematic diagram of an implementation of step 702 of an embodiment of the present application;

[0018] Fig. 9 It is a key point detection device for medical imaging according to an embodiment of the present application;

[0019] Fig.10 is a schematic diagram of a medical imaging method according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] With reference to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations and equivalents that fall within the scope of the attached claims.

[0021] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or temporal order of these elements, etc., and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0022] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0023] In the embodiments of the present application, the term "landmark" can be equivalently replaced by "key point", "key coordinate point" or "landmark point"; the term "scanned object" can be equivalently replaced by "object", "object to be scanned", "patient" or "research object", which can be a human or an animal, etc.

[0024] In the embodiments of the present application, the term "include / comprises" when used herein refers to the existence of features, whole parts, steps or components, but does not exclude the existence or addition of one or more other features, whole parts, steps or components.

[0025] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0026] In an embodiment of the present application, the obtained landmark information (for example, key point distribution information) can be applicable to various medical imaging scenarios, including but not limited to magnetic resonance imaging (MRI), computed tomography (CT), ultrasound imaging (Ultrasound Imaging), positron emission tomography (PET), single photon emission computed tomography (SPECT), PET / CT, PET / MR or any other suitable medical imaging scenario.

[0027] In the embodiments of the present application, the method, device and system of the present application are exemplarily described by taking the MRI scenario as an example. It can be understood that the contents of the embodiments of the present application are also applicable to other medical imaging scenarios.

[0028] For ease of understanding, Figure 1 FIG. 1 is a schematic diagram of a magnetic resonance imaging (MRI) system 100 according to an embodiment of the present application.

[0029] The MRI system 100 includes a scanning unit 111. The scanning unit 111 is used to perform magnetic resonance scanning on an object (eg, a human body) 170 to generate image data of a region of interest of the object 170, where the region of interest may be a predetermined anatomical site or anatomical tissue.

[0030] The operation of the MRI system 100 is controlled by an operator workstation 110, which includes an input device 114, a control panel 116, and a display 118. The input device 114 can be a joystick, a keyboard, a mouse, a trackball, a touch-activated screen, voice control, or any similar or equivalent input device. The control panel 116 can include a keyboard, a touch-activated screen, voice control, buttons, a slider, or any similar or equivalent control device. The operator workstation 110 is coupled to and communicates with a computer system 120, which enables an operator to control the generation and viewing of images on the display 118. The computer system 120 includes multiple components that communicate with each other via electrical and / or data connection modules 122. The connection module 122 can be a direct wired connection, an optical fiber connection, a wireless communication link, etc. The computer system 120 may include a central processing unit (CPU) 124, a memory 126, and an image processor 128. In some embodiments, the image processor 128 can be replaced by a medical imaging function implemented in the CPU 124. The computer system 120 may be connected to an archival media device, permanent or backup storage, or a network. The computer system 120 may be coupled to and communicate with a separate MRI system controller 130 .

[0031] The MRI system controller 130 includes a set of components that communicate with each other via an electrical and / or data connection module 132. The connection module 132 can be a direct wired connection, an optical fiber connection, a wireless communication link, etc. The MRI system controller 130 may include a CPU 131, a sequence pulse generator (also referred to as a pulse generator) 133 that communicates with the operator workstation 110, a transceiver (also referred to as an RF transceiver) 135, a memory 137, and an array processor 139.

[0032] In some embodiments, the sequence pulse generator 133 can be integrated into the resonance assembly 140 of the scanning unit 111 of the MRI system 100. The MRI system controller 130 can receive commands from the operator workstation 110, coupled to the scanning unit 111, to indicate the MRI scanning sequence to be performed during the MRI scan, so as to control the scanning unit 111 to perform the above-mentioned magnetic resonance scanning process. The MRI system controller 130 is also coupled to and communicates with the gradient drive system (also referred to as gradient drive) 150, which is coupled to the gradient coil assembly 142 to generate a magnetic field gradient during the MRI scan.

[0033] The sequence pulse generator 133 may also receive data from a physiological acquisition controller 155, which receives signals from a plurality of different sensors (e.g., electrocardiogram (ECG) signals from electrodes attached to the patient, etc.) connected to a subject or patient 170 undergoing an MRI scan. The sequence pulse generator 133 is coupled to and in communication with a scan room interface system 145, which receives signals from various sensors associated with the state of the resonance assembly 140. The scan room interface system 145 is also coupled to and in communication with a patient positioning system 147, which sends and receives signals to control the movement of the patient table to a desired position for an MRI scan.

[0034] The MRI system controller 130 provides gradient waveforms to the gradient driver system 150, which includes a G x (x direction), G y (y direction) and G z (z direction) amplifier, etc. Each G x , G y and G z The gradient amplifiers all excite corresponding gradient coils in the gradient coil assembly 142 to generate magnetic field gradients for spatially encoding MR signals during MRI scanning. The gradient coil assembly 142 is disposed within a resonance assembly 140, which also includes a superconducting magnet having superconducting coils 144, which in operation provide a static uniform longitudinal magnetic field B0 throughout a cylindrical imaging volume 146. The resonance assembly 140 also includes an RF body coil 148, which in operation provides a transverse magnetic field B1 that is substantially perpendicular to B0 throughout the cylindrical imaging volume 146. The resonance assembly 140 may also include an RF surface coil 149 for imaging different anatomical structures of a patient undergoing an MRI scan. The RF body coil 148 and the RF surface coil 149 may be configured to operate in a transmit and receive mode, a transmit mode, or a receive mode.

[0035] The x direction can also be called the frequency encoding direction or the k direction in K space. x The y direction can be called the phase encoding direction or the k direction in K space. y Direction, the z direction can be called the layer selection direction. x It can be used for frequency encoding or signal readout and is usually called frequency encoding gradient or readout gradient. y Can be used for phase encoding, usually called phase encoding gradient. G z It can be used for slice (layer) position selection to obtain K-space data. It should be noted that the slice selection direction, phase encoding direction and frequency encoding direction can be modified according to actual needs.

[0036] An MRI scanned subject or patient 170 may be positioned within a cylindrical imaging volume 146 of the resonance assembly 140. The transceiver 135 in the MRI system controller 130 generates RF excitation pulses that are amplified by an RF amplifier 162 and provided to the RF body coil 148 via a transmit / receive switch (also referred to as a T / R switch or switch) 164.

[0037] As described above, the RF body coil 148 and the RF surface coil 149 can be used to transmit RF excitation pulses and / or receive the resulting MR signals from the patient undergoing the MRI scan. The MR signals emitted by the nuclei excited in the patient's body of the MRI scan can be sensed and received by the RF body coil 148 or the RF surface coil 149 and sent back to the preamplifier 166 through the T / R switch 164. The T / R switch 164 can be controlled by a signal from the sequence pulse generator 133 to electrically connect the RF amplifier 162 to the RF body coil 148 during the transmit mode, and to connect the preamplifier 166 to the RF body coil 148 during the receive mode. The T / R switch 164 can also enable the RF surface coil 149 to be used in the transmit mode or the receive mode.

[0038] In some embodiments, the MR signals sensed and received by the RF body coil 148 or the RF surface coil 149 and amplified by the preamplifier 166 are stored as a raw k-space data array in the memory 137 for post-processing. A reconstructed magnetic resonance image can be acquired by transforming / processing the stored raw k-space data.

[0039] In some embodiments, the MR signals sensed and received by the RF body coil 148 or the RF surface coil 149 and amplified by the preamplifier 166 are demodulated, filtered, and digitized in the receive portion of the transceiver 135 and transmitted to the memory 137 in the MRI system controller 130. For each image to be reconstructed, the data is rearranged into a separate k-space data array, and each of these separate k-space data arrays is input to an array processor 139, which is operated to Fourier transform the data into an array of image data.

[0040] The array processor 139 uses a transform method, most commonly a Fourier transform, to create images from the received MR signals. These images are transmitted to the computer system 120 and stored in the memory 126. In response to commands received from the operator workstation 110, the image data may be stored in long-term storage or may be further processed by the image processor 128 and transmitted to the operator workstation 110 for presentation on the display 118.

[0041] In various embodiments, components of computer system 120 and MRI system controller 130 may be implemented on the same computer system or multiple computer systems. Figure 1 The MRI system 100 shown is for illustration purposes. Suitable MRI systems may include more, fewer, and / or different components.

[0042] The MRI system controller 130 and the image processor 128 may include a computer processor and a storage medium, respectively or in common, on which a program for predetermined data processing to be executed by the computer processor is recorded, for example, the storage medium may store a program for implementing scanning processing (e.g., scanning process, imaging sequence), image reconstruction, medical imaging, etc. For example, a program for implementing the magnetic resonance imaging method of the embodiment of the present invention may be stored. The above-mentioned storage medium may include, for example, a ROM, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, or a non-volatile memory card.

[0043] The MRI system 100 also includes a camera 180. The camera 180 is used to obtain visual and morphological information of the scanned object. Generally, the camera 180 can be installed near the examination bed, so that the image information of the scanned object can be collected to the greatest extent in a non-contact manner. The image information can be used to assist medical imaging operations. For example, the image information obtained by the camera 180 can be used to perform landmark recognition, so as to perform subsequent scanning operations based on the results of the landmark recognition.

[0044] The inventors have found that in some medical scenarios, it is necessary to cover the surface of the scanned object with an obstruction (e.g., a coil, a blanket, complex clothing, a mask, etc.). For example, in the MRI scenario described above, it is necessary to cover the surface of the scanned object with an RF surface coil 149. Since part of the scanned object is covered by the obstruction, the camera 180 cannot obtain an image within the coverage area. Therefore, when performing landmark recognition using the image obtained by the camera 180, it is easy to cause inaccurate recognition results, thereby affecting subsequent scanning results.

[0045] In response to at least one of the above problems, embodiments of the present application provide a key point detection method for medical imaging, a medical imaging method, and a medical imaging system.

[0046] The following is a description with reference to embodiments.

[0047] The present application provides a key point detection method for medical imaging. Figure 2 is a schematic diagram of a key point detection method for medical imaging according to an embodiment of the present application. Figure 2 As shown, the method includes:

[0048] Step 201: receiving an image sequence, the image sequence comprising a plurality of images of an object;

[0049] Step 202: performing key point detection on multiple images respectively to generate a key point image sequence;

[0050] Step 203: performing key point occlusion detection on the images in the image sequence; and

[0051] Step 204: Determine key point distribution information according to the key point image sequence and the key point occlusion detection result.

[0052] According to the above embodiment, key point detection is performed on multiple images in an image sequence respectively to generate a key point image sequence, and key point occlusion detection is performed on multiple images in the image sequence. The key point distribution information is determined based on the generated key point image sequence and the results of the key point occlusion detection, thereby improving the accuracy and reliability of the key point distribution information.

[0053] In other words, in the process of determining the key point distribution information of a medical image, the information of the image sequence within a period of time is taken into consideration. Compared with the method of determining the key point distribution information only based on the image acquired at a single time point, the method of this embodiment can improve the accuracy and reliability of the key point distribution information.

[0054] In some embodiments, the image sequence may be acquired by the aforementioned camera device 180. The image sequence includes a plurality of images within a period of time.

[0055] In some embodiments, the images in the image sequence may be various types of images. For example, the image may include a color image, or a depth image, or a color image and a depth image corresponding in the time dimension. The color image and the depth image corresponding in the time dimension may refer to a color image and a depth image acquired at the same time.

[0056] In some embodiments, the color image may include a plurality of pixels, and the value of each pixel may include the value of each color component, wherein the color component may include red, green, and blue (RGB) components, etc.

[0057] In some embodiments, the depth image may include multiple pixels, and the value of each pixel may include the depth value of the pixel. The depth value of the pixel may be the distance between the pixel and a reference plane, etc. The reference plane is, for example, the camera plane of the camera device 180.

[0058] In some embodiments, the image sequence may include multiple images that can show different occlusion ranges. Therefore, during key point detection, more available information can be obtained from the multiple images showing different occlusion ranges compared to one or more images showing one occlusion range.

[0059] For example, the image sequence may include at least one first image, and the key points of at least a part of the area of ​​the first image are not blocked. Therefore, even if the part of the area is blocked by an obstruction at a later time (the image with the blocked part of the area is called the second image), the key point information in the part of the area can be determined based on the information of the first image. Compared with determining the key point information in the part of the area based only on the second image, the accuracy of key point recognition can be improved.

[0060] In the following, an example is given in which an image includes a color image and a depth image corresponding to each other in the time dimension, that is, an image sequence includes a color image sequence and a depth image sequence corresponding to each other, and an exemplary description is given of a key point detection method for medical imaging in an embodiment of the present application. It is understandable that the following content is also applicable to the case in which the image only includes a color image or a depth image or other images.

[0061] In some embodiments, the color image sequence may include at least one first color image, and key points of at least a partial area of ​​the first color image are not blocked.

[0062] In some embodiments, the depth image sequence may include at least one first depth image, and key points of at least a partial area of ​​the first depth image are not blocked.

[0063] In some embodiments, in step 202, key point detection is performed on multiple images in the image sequence to generate a key point image sequence. The key point image sequence includes multiple key point images, each key point image corresponds to each image in the image sequence. The key point image includes at least one of the following information: location information, type information, confidence information, etc. of the key point.

[0064] In some embodiments, the key points may be, for example, feature points associated with body parts that can be detected in an image.

[0065] The position information of the key point may be represented by the pixel position in the image. The present application is not limited thereto, and the position information of the key point may also be represented by other methods.

[0066] Among them, the type of key point may include at least one of the following: head, chest, abdomen, neck, nose, left shoulder, right shoulder, left hip, right hip, left eye, right eye, left elbow, right elbow, left knee, right knee, left ear, right ear, left wrist, right wrist, left ankle and right ankle. The present application is not limited to this, and the key point type may also include other content.

[0067] The confidence information of the key point may be in the form of probability, for example, the probability that the real position of the key point is located at the detected position, etc. The present application is not limited thereto, and the confidence information of the key point may also be in other forms or have other meanings.

[0068] In some embodiments, a single key point or multiple key points and the positional relationship between them can be used to locate the part of the object to be imaged. For example, the region of interest to be imaged of the current object is determined, and the relative position between the object and the center of the scanning device is adjusted to align the coordinate center of the determined region of interest with the scanning center of the imaging device.

[0069] In some embodiments, in step 202, key point detection may include performing key point detection on color images in the color image sequence and depth images in the depth image sequence. The key point detection is performed on the color image to obtain a color key point image, and the key point detection is performed on the depth image to obtain a depth key point image.

[0070] In some embodiments, the key point image sequence generated by key point detection may include multiple key point images. The multiple key point images are arranged in time sequence and correspond to the images in the image sequence in sequence.

[0071] In some embodiments, each key point image may include a color key point image and a depth key point image that correspond to each other. The present application is not limited thereto, and the key point image may also be generated based on the color key point image and the depth key image that correspond to each other.

[0072] Figure 3 FIG. 2 is a schematic diagram of an implementation of step 202 of an embodiment of the present application. Figure 3 As shown, step 202 may include:

[0073] Step 301: performing key point detection on an object in each color image in a color image sequence to generate a color key point image sequence;

[0074] Step 302: performing key point detection on an object in each depth image in a depth image sequence to generate a depth key point image sequence; and

[0075] Step 303: Generate a key point image sequence according to the color key point image sequence and the corresponding depth key point image sequence.

[0076] In some embodiments, in step 301 and step 302, key point detection may be performed on the color image and the depth image in various ways to obtain a color key point image sequence and a depth key point image sequence, respectively.

[0077] For example, taking a color image in a color image sequence as an example, a neural network model can be used to perform key point detection on the color image to generate a color key point image.

[0078] The input of the neural network may be a color image, and the output of the neural network may be a color key point image including key point information. The key point information may include at least one of the following: position information, type information, confidence information, etc. of the key point. The specific method of using the neural network model to perform key point detection can refer to the relevant technology, which will not be described in detail here.

[0079] Similarly, the above method may be used to generate a depth key point image based on the depth image.

[0080] In some embodiments, in step 303, in the process of generating a key point image sequence based on a color key point image sequence and a corresponding depth key point image sequence: for a color key point image in the color key point image sequence and a corresponding depth key point image in the depth key point image sequence, information of a corresponding key point in the corresponding key point image in the key point image sequence is determined based on the confidence of the key point in the color key point image and the confidence of the corresponding key point in the depth key point image.

[0081] For example, among the corresponding key points in the color key point image and the depth key point image, a key point with a large confidence is selected, and the information of the key point (for example, position information) is used as the information of the key point in the corresponding key point image.

[0082] The color key point image corresponds to the depth key point image, which means that the color key point image and the depth key point image are acquired at the same time, that is, the color key point image and the depth key point image are aligned in time. The key point image corresponding to the color key point image or the depth key point image may refer to a key point image generated based on the color key point image or the depth key point image.

[0083] Among them, corresponding key points may refer to key points of the same type, for example, the nose key point in the color key point image, the nose key point in the depth key point image, and the nose key point in the key point image correspond to each other.

[0084] In some embodiments, in step 202, preprocessing may also be included. The color image sequence and the depth image sequence may be generated after preprocessing the color stream and the depth stream. The preprocessing may include normalization, resizing, data cleaning, and other operations. Thus, the accuracy of key point detection can be improved.

[0085] In some embodiments, in step 202, post-processing may also be included. After key point detection, the detection results of the key point detection may be post-processed to obtain a color key point image sequence and a depth key point image sequence. The post-processing may include key point denoising. Thus, the accuracy of the key point image sequence can be improved. The present application is not limited to this. For example, in step 202, when the results of directly detecting key points using the original color image and the depth image are accurate enough, pre-processing or post-processing may not be included.

[0086] Figure 4 It is a schematic diagram of the process of step 202 of the embodiment of the present application. Figure 4 The dashed boxes represent data, and the solid boxes represent processing. Figure 4 As shown, in step 202: the color stream and the depth stream are preprocessed respectively (step 401 and step 402), and a color image sequence and a depth image sequence are generated. The color image sequence includes N color images, and the depth image sequence includes N depth images. The color images with the same number correspond to the depth images, that is, the i-th color image corresponds to the i-th depth image, wherein N is greater than or equal to 1, and i is a positive integer less than or equal to N.

[0087] Key point detection is performed on each color image and each depth image (step 403 and step 404), and the detection results of the key point detection are post-processed (step 405 and step 406) to generate a color key point image sequence and a depth key point image sequence. The color key point image sequence includes N color key point images, and the depth key point image sequence includes N depth key point images. Color key point images with the same number correspond to depth key point images, color key point images with the same number correspond to color images, and depth key point images with the same number correspond to depth images.

[0088] The depth key point image sequence and the color key point image sequence are merged (step 407) to generate a key point image sequence, wherein the key point image sequence includes N key point images, and the key point images, color image key point images, and depth key point images with the same number correspond to each other.

[0089] In step 407, for the same type of key points in the i-th color key point image and the i-th depth key point image, a key point with a large confidence is selected as the key point of the type in the i-th key point image. For example, the confidence of the nose key point in the i-th color key point image is 80%, and the confidence of the nose key point in the i-th depth key point image is 60%. The nose key point in the i-th color key point image is used as the nose key point in the i-th key point image, that is, the position information of the nose key point in the i-th key point image is the position information of the nose key point in the i-th color key point image.

[0090] In some embodiments, in step 203, key point occlusion detection is performed on images in the image sequence. For example, key point occlusion detection may be performed on color images in the color image sequence and depth images in the depth image sequence, and the key point detection result is used as occlusion information.

[0091] The key point occlusion detection may include detecting the occlusion state of the key point and time information when the occlusion state changes.

[0092] Figure 5 Schematic diagram of the key point occlusion state of the embodiment of the present application. Figure 5 As shown in FIG. 1 , the neural network model can detect 15 key points. When a part of the detected object is blocked by an occluder, the key points in the part are blocked; the key points in other parts are not blocked.

[0093] For example, Figure 5 As shown, the abdominal area of ​​the detected object is blocked by the blanket, and the abdominal key point C, the left hip key point B and the right hip key point A in the area are blocked; other key points are not blocked.

[0094] The occlusion information can be used to indicate the occlusion state of the key points of the object and the time information when the occlusion state changes. Occlusion detection can be performed in various ways such as image recognition and neural networks. For example, taking a color image as an example, the range of the object in the color image that is occluded can be determined by image recognition, and then the key points within the occluded range can be determined based on the results of the key point detection of the color image by the neural network. Alternatively, the key points of the object in the color image that are occluded can be determined directly based on the results of the key point detection of the color image by the neural network (for example, undetected key points or key points with low confidence are regarded as occluded key points). The input of the neural network can be a color image, and the output can be information on the type of occluded key points, etc. Among them, the specific methods for key point detection and determining the occluded range / key points can refer to the relevant technology, which will not be described in detail here.

[0095] In some embodiments, the occlusion state of the key points of the object and the time information of the change of the occlusion state can be represented in various forms. For example, the occlusion information may include the occlusion state of each key point at each time. Table 1 shows an example of occlusion information.

[0096] Table 1

[0097] Moment 1 … Moment i … Time N Key Point 1 1 … 0 … 1 … … … … … … Key Points 1 … 1 … 0 … … … … … … Key Points 1 … 0 … 0

[0098] In Table 1, N is the number of color images or depth images, M is the total number of key points that the neural network model can detect, 0 in the table indicates that the key point is occluded at a certain moment (frame), and 1 in the table indicates that the key point is not occluded at a certain moment (frame).

[0099] For another example, the occlusion information may include the occlusion status of each key point at the last moment. For the key point that is occluded at the last moment, the occlusion information also includes the time when the occlusion status of the key point changes from unoccluded to occluded. If there are multiple times, only the last time may be indicated.

[0100] For example, in the occlusion information, only the occlusion status of each key point in the Nth key point image is indicated. Taking an occluded key point in the Nth key point image as an example, if the key point is not occluded in the i-th image and is occluded in the i+1-th image, the occlusion information indicates the time corresponding to the i-th image or the time corresponding to the i+1-th image for the key point; or, if the key point is not occluded in the i-th image and is occluded in the i+1-th image, and is not occluded in the j-th image and is occluded in the j+1-th image, j is greater than i, then the occlusion information indicates the time corresponding to the j-th image or the time corresponding to the j+1-th image for the key point. The present application is not limited to this, and the occlusion information may also be other forms of information, as long as it can indicate the occlusion status and time information of the key point.

[0101] In some embodiments, in step 204, key point distribution information is determined based on the key point image sequence acquired in step 202 and the result of key point occlusion detection acquired in step 203 (ie, occlusion information). Figure 6 is a schematic diagram of an implementation of step 204 of an embodiment of the present application, such as Figure 6 As shown, step 204 may include:

[0102] Step 601: determining a key point occlusion image in a key point image sequence;

[0103] Step 602: In the key point occlusion image, determine a first key point that is occluded and a second key point that is not occluded according to the occlusion status of the key point;

[0104] Step 603: determining an unobstructed image of the key point in the key point image sequence according to the time information of the change of the occlusion state of the first key point;

[0105] Step 604: Determine key point distribution information according to the key point unobstructed image and the key point obstructed image.

[0106] For the first key point that is occluded in the key point occluded image, a key point unoccluded image is determined in the key point image sequence, and key point distribution information is generated based on the key point occluded image and the key point unoccluded image, which can improve the accuracy and reliability of the key point distribution information.

[0107] In some embodiments, in step 601, the key point occlusion image may be the last key point image in the key point image sequence, for example, the Nth key point image. Since the last key point image is the key point image closest to the current time, the key point distribution information determined based on the last key point image can more accurately reflect the current position information of the object. The present application is not limited to this, and the key point occlusion image may also be any one or more key point images in the key point image sequence.

[0108] In some embodiments, in a key point occluded image, at least part of the key points are occluded, so that the key points in the image are not detected or the detected key points have a lower confidence.

[0109] In some embodiments, in step 602, the first key point that is blocked and the second key point that is not blocked in the key point occlusion image may be determined according to the result of the key point occlusion detection.

[0110] In some embodiments, in step 603, for each blocked first key point in the key point blocked image, a key point unblocked image corresponding to the first key point is determined. For example, in the key point blocked image, the chest key point and the abdomen key point are blocked, and the key point unblocked image for the chest key point is determined in the key point image sequence based on the chest key point, and the key point unblocked image for the chest key point is determined in the key point image sequence based on the abdomen key point.

[0111] In some embodiments, in the key point unoccluded image, the key point corresponding to the first key point is not occluded, so that the key point in the image is detected or the detected key point has a higher confidence.

[0112] In some embodiments, the key point unobstructed image is the following key point image in the key point image sequence: within the key point unobstructed image, the key point of the same type as the first key point is not obstructed, and the key point unobstructed image is closest to the key point occluded image in the time dimension of the key point image sequence.

[0113] For example, there are 10 key point images in the key point image sequence, and the key point occlusion image is the 10th key point image. In the 10th key point image, the chest key point is occluded. According to the time information when the occlusion state of the chest key point changes, for example, 1111100000 (i.e., the chest key point is not occluded in the 1st to 5th images or moments, and is occluded in the 6th to 10th images or moments), the 5th key point image is selected as the key point unoccluded image of the chest key point.

[0114] For another example, when the time information when the occlusion state of the chest key point changes is 111110110 (i.e., the chest key point is not occluded in the 1st to 5th, 8th to 9th images or moments, and is occluded in the 6th and 10th images or moments), the 9th key point image is selected as the key point unoccluded image of the chest key point.

[0115] As mentioned above, for a first key point, the key point unobstructed image is a key point image determined in the above manner. The present application is not limited thereto, and for a first key point, the key point unobstructed image may also be multiple key point images.

[0116] For example, the key point unobstructed image may be: a key point of the same type as the first key point that is not obstructed, and a key point image whose distance from the key point obstructed image in the time dimension is less than a preset distance. For example, when the time information of the change in the obstruction state of the chest key point is 111110110, the 8th and 9th key point images, which are not obstructed for the chest key point and whose time interval with the key point obstructed image (the 10th key point image) is less than 3 images, are selected as the key point unobstructed images for the chest key point.

[0117] In some embodiments, in step 604, key point distribution information may be generated in various ways according to the key point unoccluded image and the key point occluded image. Figure 7 FIG. 6 is a schematic diagram of an implementation of step 604 of an embodiment of the present application. Figure 7 As shown, step 604 may include:

[0118] Step 701: determining a third key point that is not blocked in the key point unblocked image, where the third key point is of the same type as the first key point that is blocked;

[0119] Step 702: Generate key point distribution information according to the third key point in the key point unoccluded image and the unoccluded second key point in the key point occluded image.

[0120] In some embodiments, in step 702 , key point distribution information may be determined in various ways. Figure 8 FIG. 7 is a schematic diagram of an implementation method of step 702 of an embodiment of the present application. Figure 8 As shown, step 702 may include:

[0121] Step 801: using the position information of the third key point in the key point unobstructed image to update the position information of the first key point in the key point obstructed image; and

[0122] Step 802: Generate key point distribution information according to the updated position information of the first key point and the position information of the second key point.

[0123] Since the third key point in the key point unobstructed image is not obstructed, the reliability of the position information of the third key point is higher. The position information of the first key point in the key point occluded image is replaced by the position information of the third key point in the key point unobstructed image, and the key point distribution information is generated based on the updated position information of the first key point, which can improve the accuracy of the key point distribution information.

[0124] In some embodiments, the key point distribution information may include updated position information of the first key point and position information of the second key point.

[0125] The present application is not limited thereto, and the position information of the first key point may also be updated in other ways. For example, the position information of the first key point and the position information of the third key point may be weighted, for example, weighted according to the confidence level, and the weighted result may be used as the updated position information of the first key point, and so on.

[0126] In addition, when the key point unobstructed image includes multiple key point images, the position information of the first key point can also be updated according to the position information of multiple third key points in the multiple key point images. For example, the position information of the multiple third key points can be weighted, or the position information of the first key point and the position information of the multiple third key points can be weighted, and the weighted result can be used as the updated position information of the first key point, and so on.

[0127] According to the above embodiment, key point detection is performed on multiple images in an image sequence respectively to generate a key point image sequence, and key point occlusion detection is performed on multiple images in the image sequence. The key point distribution information is determined based on the generated key point image sequence and the results of the key point occlusion detection, thereby improving the accuracy and reliability of the key point distribution information.

[0128] In other words, in the process of determining the key point distribution information of a medical image, the information of the image sequence within a period of time is taken into consideration. Compared with the method of determining the key point distribution information only based on the image acquired at a single time point, the method of this embodiment can improve the accuracy and reliability of the key point distribution information.

[0129] The embodiment of the present application also provides a key point detection device for medical imaging, and the contents that are the same as the above embodiment are not repeated here.

[0130] Fig. 9 is a schematic diagram of a key point detection device for medical imaging according to an embodiment of the present application. Fig. 9 As shown, the key point detection device 900 for medical imaging includes:

[0131] A receiving unit 901 receives an image sequence, wherein the image sequence includes a plurality of images of an object;

[0132] A key point detection unit 902, which performs key point detection on the multiple images respectively to generate a key point image sequence;

[0133] an occlusion detection unit 903, which performs key point occlusion detection on the images in the image sequence; and

[0134] The determining unit 904 determines key point distribution information according to the key point image sequence and the key point occlusion detection result.

[0135] In some embodiments, the image sequence includes at least one first image, and key points of at least a partial area of ​​the first image are not blocked.

[0136] In some embodiments, the image sequence includes a color image sequence and a depth image sequence that correspond to each other, and the key point detection includes performing key point detection on color images in the color image sequence and depth images in the depth image sequence.

[0137] In some embodiments, the color image sequence includes at least one first color image, and key points of at least a partial area of ​​the first color image are not blocked.

[0138] In some embodiments, the depth image sequence includes at least one first depth image, and key points of at least a partial area of ​​the first depth image are not blocked.

[0139] In some embodiments, the key point image sequence includes a plurality of key point images, wherein each of the key point images includes a color key point image and a depth key point image that correspond to each other.

[0140] In some embodiments, the key point occlusion detection includes detecting the occlusion state of the key point and time information of the change of the occlusion state.

[0141] In some embodiments, the determination unit 904 determines a key point occluded image in the key point image sequence; in the key point occluded image, determines a first key point that is occluded and a second key point that is not occluded according to the occlusion state of the key point; determines a key point unoccluded image in the key point image sequence according to the time information of the change in the occlusion state of the first key point; and determines the key point distribution information according to the key point unoccluded image and the key point occluded image.

[0142] In some embodiments, the keypoint occlusion image is the last image in the keypoint image sequence.

[0143] In some embodiments, in the key point unoccluded image, key points of the same type as the first key point are not occluded, and in the time dimension of the key point image sequence, the key point unoccluded image is closest to the key point occluded image.

[0144] In some embodiments, the determination unit 904 determines a third key point that is not obstructed in the key point unobstructed image, and the third key point is of the same type as the first key point that is obstructed; and generates the key point distribution information based on the third key point in the key point unobstructed image and the second key point that is not obstructed in the key point occluded image.

[0145] In some embodiments, the determination unit 904 uses the position information of the third key point in the key point unobstructed image to update the position information of the first key point in the key point occluded image; and generates the key point distribution information based on the updated position information of the first key point and the position information of the second key point.

[0146] It is worth noting that the above only describes the components or modules related to the present application, but the present application is not limited thereto. The key point detection device 900 for medical imaging may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.

[0147] To keep it simple, Fig. 9 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0148] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0149] According to the above embodiment, key point detection is performed on multiple images in an image sequence respectively to generate a key point image sequence, and key point occlusion detection is performed on multiple images in the image sequence. The key point distribution information is determined based on the generated key point image sequence and the results of the key point occlusion detection, thereby improving the accuracy and reliability of the key point distribution information.

[0150] In other words, in the process of determining the key point distribution information of a medical image, the information of the image sequence within a period of time is taken into consideration. Compared with the method of determining the key point distribution information only based on the image acquired at a single time point, the method of this embodiment can improve the accuracy and reliability of the key point distribution information.

[0151] The embodiment of the present application also provides a medical imaging method. Fig.10 FIG. 1 is a schematic diagram of a medical imaging method according to an embodiment of the present application. Fig.10 As shown, the medical imaging method includes:

[0152] Step 1001: determining key point distribution information according to an image sequence; and

[0153] Step 1002: Perform a scanning operation according to the determined key point distribution information.

[0154] In step 1001, key point distribution information may be determined according to the key point detection method for medical imaging described in the above embodiments, the contents of which are incorporated herein and will not be further described.

[0155] In some embodiments, in step 1002, performing a scanning operation according to the determined key point distribution information may include positioning the scanned object according to the key point distribution information. For the positioning content, reference may be made to the relevant content or related technology in the aforementioned embodiments, and the content thereof is incorporated herein and will not be further described.

[0156] In some embodiments, step 1002 may also include performing a scanning operation on the positioned scanning object.

[0157] According to the above embodiment, since the key point detection method for medical imaging described in the above embodiment is used to determine the key point distribution information, the key point distribution information has high accuracy and reliability. When a scanning operation is performed based on the key point distribution information, the accuracy and reliability of the scanning result can be guaranteed.

[0158] The present application also provides a medical imaging system. The medical imaging system is composed of Figure 1 As shown, the repeated parts will not be repeated.

[0159] In some embodiments, Figure 1 The difference between the aforementioned medical imaging system and the aforementioned medical imaging system is that the controller 130 is configured to execute the aforementioned key point detection method for medical imaging, that is, receiving an image sequence, the image sequence comprising a plurality of images of an object; performing key point detection on the plurality of images respectively to generate a key point image sequence; performing key point occlusion detection on the images in the image sequence; and determining key point distribution information according to the key point image sequence and the results of the key point occlusion detection.

[0160] In some embodiments, the controller 130 (which may also be a processor) includes a computer processor and a storage medium, on which a program for predetermined data processing to be executed by the computer processor is recorded. For example, the storage medium may store programs for implementing scanning processing (for example, including waveform design / conversion, etc.), image reconstruction, medical imaging, etc. For example, a program for implementing a key point detection method for medical imaging according to an embodiment of the present invention may be stored. The key point detection method for medical imaging includes: receiving an image sequence, wherein the image sequence includes multiple images of an object; performing key point detection on the multiple images respectively to generate a key point image sequence; performing key point occlusion detection on the images in the image sequence; and determining key point distribution information based on the key point image sequence and the results of the key point occlusion detection. The specific implementation method is as described above and will not be repeated here. The above storage medium may include, for example, a ROM, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, or a non-volatile memory card.

[0161] The present application is not limited thereto, and the aforementioned key point detection method for medical imaging may also be executed in other processors in the MRI system 100, or executed by a cloud processor.

[0162] In some embodiments, the scanning component may perform a scanning operation according to the key point distribution information determined by the controller 130. The scanning component may include a positioning component, which may position the scanned object according to the key point distribution information. The positioning component is, for example, the patient positioning system 147 of the MRI system 100.

[0163] In some embodiments, the scanning component may further include a scanning unit 111, and the scanning unit 111 performs a scanning operation on the positioned scanning object.

[0164] According to the above embodiment, key point detection is performed on multiple images in an image sequence respectively to generate a key point image sequence, and key point occlusion detection is performed on multiple images in the image sequence. The key point distribution information is determined based on the generated key point image sequence and the results of the key point occlusion detection, thereby improving the accuracy and reliability of the key point distribution information.

[0165] In other words, in the process of determining the key point distribution information of a medical image, the information of the image sequence within a period of time is taken into consideration. Compared with the method of determining the key point distribution information only based on the image acquired at a single time point, the method of this embodiment can improve the accuracy and reliability of the key point distribution information.

[0166] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in a medical imaging system, the program enables the computer to execute the key point detection method for medical imaging described in the aforementioned embodiment in the medical imaging system.

[0167] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the key point detection method for medical imaging described in the aforementioned embodiment in a medical imaging system.

[0168] The above devices and methods of the present application can be implemented by hardware, or by hardware combined with software. The present application relates to such a computer-readable program, which, when executed by a logic component, enables the logic component to implement the above-mentioned devices or components, or enables the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0169] The method / device described in conjunction with the embodiments of the present application may be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to the various steps shown in the figure, respectively. These hardware modules may be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).

[0170] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if a device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0171] For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present application. For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0172] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.

Claims

1. A key point detection method for medical imaging, characterized in that: The method comprises: receiving an image sequence, the image sequence comprising a plurality of images of an object; Performing key point detection on the multiple images respectively to generate a key point image sequence; Performing key point occlusion detection on images in the image sequence; and Key point distribution information is determined according to the key point image sequence and the key point occlusion detection result.

2. The method according to claim 1, characterized in that The image sequence includes at least one first image, and key points of at least a partial area of ​​the first image are not blocked.

3. The method according to claim 1 or 2, characterized in that: The image sequence includes a color image sequence and a depth image sequence that correspond to each other, and the key point detection includes performing key point detection on the color images in the color image sequence and the depth images in the depth image sequence.

4. The method according to claim 3, characterized in that The color image sequence includes at least one first color image, and key points of at least a partial area of ​​the first color image are not blocked.

5. The method according to claim 3, characterized in that: The depth image sequence includes at least one first depth image, and key points of at least a partial area of ​​the first depth image are not blocked.

6. The method according to claim 1, characterized in that The key point image sequence includes a plurality of key point images, wherein each of the key point images includes a color key point image and a depth key point image that correspond to each other.

7. The method according to claim 1, characterized in that The key point occlusion detection includes detecting the occlusion state of the key point and time information of the change of the occlusion state.

8. The method according to claim 7, characterized in that Determining the key point distribution information according to the key point image sequence and the key point occlusion detection result includes: Determining a key point occlusion image in the key point image sequence; In the key point occlusion image, determining a first key point that is occluded and a second key point that is not occluded according to the occlusion state of the key point; Determining a key point unobstructed image in the key point image sequence according to time information of a change in the occlusion state of the first key point; and The key point distribution information is determined according to the key point unobstructed image and the key point obstructed image.

9. The method according to claim 8, characterized in that The key point occlusion image is the last key point image in the key point image sequence.

10. The method according to claim 8, characterized in that In the key point unoccluded image, key points of the same type as the first key point are not occluded, and in the time dimension of the key point image sequence, the key point unoccluded image is closest to the key point occluded image.

11. The method according to claim 8, characterized in that Determining the key point distribution information according to the key point unobstructed image and the key point obstructed image includes: Determine a third key point that is not blocked in the key point unblocked image, where the third key point is of the same type as the first key point that is blocked; The key point distribution information is generated according to the third key point in the key point unoccluded image and the second key point that is not occluded in the key point occluded image.

12. The method according to claim 11, characterized in that Generating the key point distribution information according to the third key point in the key point unblocked image and the second key point that is not blocked in the key point blocked image, comprising: Using the position information of the third key point in the key point unoccluded image to update the position information of the first key point in the key point occluded image; and The key point distribution information is generated according to the updated position information of the first key point and the position information of the second key point.

13. A medical imaging method comprising: Determine key point distribution information according to the method described in any one of claims 1 to 12; as well as A scanning operation is performed according to the determined key point distribution information.

14. The method according to claim 13, characterized in that The scanning operation performed according to the determined key point distribution information includes: The scanned object is positioned according to the key point distribution information.

15. A medical imaging system, characterized in that: The system comprises: A controller configured to execute the key point detection method for medical imaging according to any one of claims 1 to 12; and A scanning component performs a scanning operation according to key point distribution information determined by a controller.

16. The medical imaging system according to claim 15, characterized in that The scanning component comprises: A positioning component is used to position the scanned object according to the key point distribution information.

Images