Multi-mode image visualization for stroke detection

Abstract

A combined near infrared spectroscopy (NIRS) and ultrasound imaging system acquires NIRS image data and ultrasound image data of the brain and reconstructs and fuses the information into a three-dimensional data set using the geometric information of the ultrasound transducers and NIRS transmitters and sensors. And extracting the NIRS and information indicating the related area to be analyzed in the ultrasonic image from the spatial related data. These regions may be specific anatomical regions, such as vascular structures identified in transcranial ultrasound, and may also be regions where the acquired data may indicate stroke. The detection algorithm of the related area can be based on a neural network and a deep learning method.

Description

Technical Field

[0001] The present invention relates to medical diagnostic imaging systems, and more particularly to the use of ultrasound imaging and near infrared spectroscopy in combination for the detection and diagnosis of stroke. Background Art

[0002] Stroke is one of the most debilitating conditions known to medicine. Blockage of blood flow to the brain (ischemic stroke) or rupture of a blood vessel in the brain (hemorrhagic stroke) can rapidly lead to paralysis or death. Attempts to recanalize blocked blood vessels with clot-busting drug therapy, such as treatment with tissue plasminogen activator (tPA), have been reported to result in asymptomatic intracerebral bleeding in some cases. Advances in the diagnosis and treatment of this debilitating condition are an ongoing subject of medical research.

[0003] U.S. Patent 8,211,023 (Swan et al.) describes an ultrasound diagnostic system and method that allows clinicians to transcranially observe areas of cerebral blood vessels where blood clots may be present. Both two-dimensional and three-dimensional imaging can be used. Preferably, the imaging of the blood vessels is enhanced by the administration of microbubbles. If the flow conditions in the blood vessel indicate the presence of a partial or complete blockage caused by a blood clot, a focused or pencil beam is directed to the blockage site, and the blood clot is broken up by the vibration and / or rupture of the microbubbles, a process called acoustic thrombolysis. In some cases, the ruptured microbubbles may also release encapsulated thrombolytic drugs. The patent also describes monitoring changes in cranial and cerebral blood vessels through ultrasound imaging, which changes can indicate the recurrence of the blockage, thereby alerting medical rescue personnel to the recurrence.

[0004] Another imaging modality that can be used for stroke detection is near infrared spectroscopy or NIRS. NIRS is based on the same physical principle as pulse oximetry, which involves irradiating blood in the body with near infrared wavelengths that are sensitive to the color of oxygenated blood. The intensity of the light returning at the detected wavelength can measure the oxygen content of the irradiated blood. In a cranial NIRS system, a headset is worn by the patient with a number of transmitting laser diodes and highly sensitive photodiodes. The headset transmits infrared frequency light to the patient's head through the transmitting laser diodes and receives non-absorbed light reflected to the photodiodes. At the wavelength used, infrared light is able to penetrate the cerebral cortex for several centimeters (e.g., 2-3 cm), and a photosensitive detector receives the reflected light to sense the amount of oxygenated or deoxygenated blood at the irradiated location. The amount of oxygenated or deoxygenated blood and their relative proportions at different locations in the cerebral cortex can then be assessed to determine if there is perfusion or blood flow deficiency.

[0005] When diagnosing and treating a stroke, it is important to be able to quickly determine whether a stroke has occurred, where in the brain the stroke is, and what type of stroke it is so that appropriate treatment can be quickly initiated. As discussed above, applying the wrong treatment, such as applying tPA when a patient has had a hemorrhagic stroke, can have serious consequences. One way to increase diagnostic certainty is to use different diagnostic modalities for diagnosis, so that the results of one diagnosis can confirm or question the results of another diagnosis. U.S. Patent Publication 2019 / 0231316 (Sutton et al.) describes a multimodal method for diagnosing and treating myocardial infarction, in which the results of a digital ECG (electrocardiogram) diagnosis are used to help locate blood clots for ultrasound imaging and subsequent sonic thrombolytic therapy. Accordingly, it would be desirable to combine both ultrasound imaging and NIRS imaging for stroke detection and subsequent treatment. Summary of the invention

[0006] According to the principles of the present invention, a combined ultrasound cranial imaging system and near infrared spectroscopy system for diagnosing stroke is described. Both the ultrasound system and the NIRS system generate cranial analysis data in a common spatial coordinate system. In this way, the data can be merged together or fused into a single two-dimensional or three-dimensional brain image map in a common analysis framework (such as spatially corresponding images). The pathology revealed by the image data of one mode should be spatially correlated with the pathology determined by the other mode, allowing the clinician to confidently implement an appropriate treatment plan. For example, a possible blocked blood vessel detected in an ultrasound image can be confirmed by the lack of oxygenated blood in the cranial region downstream of the suspected blockage. Such a system can use a common transducer / sensor headset that can be quickly applied to the patient to generate data from both modes in a common coordinate system, thereby quickly performing diagnosis and treatment.

[0007] A multi-modal imaging system can diagnose brain abnormalities (such as signs of stroke), and the multi-modal imaging system includes: a NIRS system, which is suitable for acquiring NIRS signals from the brain and using the acquired NIRS signals to generate NIRS image data; an ultrasound imaging system, which is suitable for acquiring ultrasound signals from the brain and using the acquired ultrasound signals to generate ultrasound image data; an image processor, which is suitable for aligning the NIRS image data and the ultrasound image data to a common spatial coordinate system; and an image display, which is suitable for displaying the spatially correlated NIRS image data and the ultrasound image data.

[0008] A multimodal method of the present invention is used to diagnose brain abnormalities (such as signs of stroke), and the multimodal method includes: operating a NIRS system, which is suitable for acquiring NIRS signals from the brain and using the acquired NIRS signals to generate NIRS image data; operating an ultrasound imaging system, which is suitable for acquiring ultrasound signals from the brain and using the acquired ultrasound signals to generate ultrasound image data; operating an image processor, which is suitable for aligning the NIRS image data and the ultrasound image data to a common spatial coordinate system; and operating an image display, which is suitable for displaying spatially correlated NIRS image data and ultrasound image data.

[0009] According to the present invention, a method for diagnosing suspicious areas of the body using multiple imaging modes includes: acquiring image data of the body area using a first imaging mode; acquiring image data of the body area using a second imaging mode, wherein the image data of the two imaging modes are related to a common coordinate system; generating an image of the body area using the image data of each mode; detecting suspicious pathological areas in the images of the two modes; merging the image data of the detected areas into a three-dimensional data set; selecting a suspicious pathological area; displaying images of one or two modes of the selected suspicious pathological area; and displaying diagnostic data related to the suspicious pathology. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the attached picture:

[0011] Figure 1 An ultrasonic diagnostic imaging system constructed in accordance with the principles of the present invention is illustrated in block diagram form.

[0012] Figure 2 A cranial head mounted device for use with a hand held ultrasound sensor probe is described.

[0013] Figure 3 A head mounted device for NIRS transmitters and sensors and ultrasound imaging probes is described.

[0014] Figure 4 A flat ultrasound transducer probe for use with an ultrasound / NIRS headset is illustrated.

[0015] Figure 5 An ultrasound scan of the brain performed with two contralateral transducer arrays is illustrated.

[0016] Figure 6 An ultrasound image of the brain is illustrated.

[0017] Figure 7 A cranial NIRS image is illustrated.

[0018] Figure 8An image display showing spatially corresponding NIRS data and ultrasound image data in accordance with the principles of the present invention is described.

[0019] Fig. 9 Schematic diagram of the concept of combining spatially aligned ultrasound and NIRS image data according to the principles of the present invention.

[0020] Fig.10 A three-dimensional memory and volume rendering processor for producing a three-dimensional cerebrovascular map containing ultrasound and NIRS image data in accordance with the principles of the present invention is described.

[0021] Fig.11a and Fig.11b is a flow chart illustrating a method for multimodal diagnosis of a suspicious area of ​​the body using ultrasound imaging and NIRS imaging. DETAILED DESCRIPTION

[0022] First reference Figure 1 , which shows in block diagram form a combined ultrasound and NIRS system constructed according to the principles of the present invention. Two transducer arrays 10a and 10b are provided for transmitting ultrasound waves and receiving echo information. In this example, the arrays are shown as two-dimensional arrays of transducer elements (matrix arrays) capable of scanning a volumetric region and providing three-dimensional image data for imaging. The transducer arrays are coupled to microwave beamformers (formers) 12a and 12b, which control the transmission and reception of signals by the array elements. The microwave beamformers are also capable of at least partially beamforming the signals received by groups or "patches" of transducer elements, as described in U.S. Patents 5,997,479 (Savord et al.), 6,013,032 (Savord) and 6,623,432 (Powers et al.). Multiplexer 14 transmits signals to and from the microwave beamformers in a time-interleaved signal. The multiplexer is coupled to a transmit / receive (T / R) switch 16 that switches between transmit and receive and protects the main beamformer 20 from high energy transmit signals. Transmission of ultrasound beams by the transducer arrays 10a and 10b under the control of the microbeamformers 12a and 12b is directed by a transmit controller 18 that is coupled to the T / R switch, receives user-operated input from a user interface or control panel 38, and controls the steering direction and focus of beams transmitted from and to the array transducers in accordance with system control settings.

[0023] The partial beamformed signals generated by the microbeamformers 12a and 12b are coupled to the main beamformer 20, where the partial beamformed signals from the individual element patches (element units) are combined into a fully beamformed signal. For example, the main beamformer 20 may have 128 channels, each receiving a partial beamformed signal from a patch consisting of 12 transducer elements. In this way, the signals received by more than 1,500 transducer elements of the two-dimensional array can effectively be combined into one beamformed signal.

[0024] The beamformed signals are coupled to a fundamental / harmonic signal separator 22. The function of the separator 22 is to separate the linear and nonlinear signals so that the strong nonlinear echo signals returned from the microbubbles can be identified. The separator 22 can operate in a variety of ways, such as bandpass filtering the received signals at the fundamental and harmonic frequencies, or using a method called pulse inversion harmonic separation. A suitable fundamental / harmonic signal separator is shown and described in International Patent Publication WO2005 / 074805 (Bruce et al.). The separated signals are coupled to a signal processor 24, where they can undergo additional enhancement processing such as speckle removal, signal recombination and noise elimination.

[0025] The processed signals are coupled to a B-mode processor 26 and a Doppler processor 28. The B-mode processor 26 uses amplitude detection to image body structures (such as muscles, tissues, and blood vessels). B-mode images of body structures can be formed in harmonic mode or fundamental mode. Both types of signals are returned by tissues and microbubbles in the body, and the harmonic return of microbubbles allows the tissues and blood flow perfused by microbubbles to be clearly separated in the image. The Doppler processor processes the temporally different signals from the moving tissues and blood flow to detect the movement of materials including microbubbles in the image field. The structural signals and motion signals generated by these processors are coupled to a scan converter 32 and a volume renderer 34, which generate image data of tissue structure, blood flow, or a combination of both features. The scan converter converts the echo signals in polar coordinates into image signals in a desired image format, such as a sector image in Cartesian coordinates. The volume renderer 34 converts the three-dimensional data set into a projected three-dimensional image viewed from a given reference point (fiducial point), as described in U.S. Patent No. 6,530,885 (Entrekin et al.). As described therein, when the reference point of the rendering changes, the three-dimensional image appears to rotate, or the three-dimensional image is viewed from different directions, which is called dynamic parallax. This image manipulation is controlled by the user, as shown by the display control line between the user interface 38 and the volume renderer 34. In addition, three-dimensional volume rendering achieved by planar images in different image planes is described, a technique known as multi-planar reconstruction. As described in U.S. Patent No. 6,723,050 (Dow et al.), the volume renderer 34 can operate on image data in rectangular coordinates or polar coordinates. The two-dimensional or three-dimensional image is coupled from the scan converter and volume renderer to the image processor 30 for further enhancement, buffering and temporary storage for display on the image display 40.

[0026] A graphics processor 36 is also coupled to the image processor 30 for generating graphic overlays to be displayed with the ultrasound image. These graphic overlays may contain standard identification information (e.g., patient name), date and time of the image, imaging parameters (e.g., depth), etc., and may also generate graphic overlays of beam vectors steered (manipulated) by the user, such as Figure 5 As shown. To this end, the graphics processor receives input from the user interface 38. The user interface is also coupled to the transmit controller 18 to control the transducer arrays 10a and 10b to generate ultrasound signals, thereby controlling the images produced by the transducer arrays and the treatment applied. The transmit parameters controlled in response to user adjustments include MI (mechanical index), which controls the peak pressure of the transmit wave. For example, a low MI can be used for microbubble harmonic imaging and a high MI can be used for acoustic thrombolysis therapy. Other transmit parameters can control the steering of the transmit beam for image positioning and / or treatment beam positioning (steering).

[0027] According to the principles of the present invention, a near infrared spectroscopy (NIRS) system 50 is coupled to an image processor so that NIRS image data can be coupled to the system for observation and combination with ultrasound information. The NIRS system 50 can be composed of a commercially available system, such as the NIRScout system of NIRx Medizintechnik GmbH, Berlin, Germany, or the Hitachi ETG 4100fNIRS system of Philips Healthcare, Eindhoven, the Netherlands. NIRS is based on the strong chromogenicity or light absorption characteristics of hemoglobin molecules. The NIRS system emits near infrared light of a wavelength that can be absorbed by hemoglobin to the brain, and the signal in the form of non-absorbed light is reflected back to one or more photodiodes and detected at the photodiodes, thereby estimating the amount of hemoglobin at the irradiated position. Since oxygenated hemoglobin and deoxygenated hemoglobin have different absorption of light, it is preferred to use two different wavelengths of light, i.e., 695nm-760nm and 830nm-900nm, so that the ratio of oxygenated hemoglobin to deoxygenated hemoglobin and the level of cerebral arteriovenous oxygen saturation can be calculated. The data generated by NIRS systems are usually calibrated in arbitrary units because the data are derived from optical density units and converted to dimensionless units by mathematical methods. Therefore, NIRS data values ​​are usually used for comparison, such as monitoring the trend of data values ​​at a certain location that may decrease over time, or comparing the differences in NIRS data values ​​measured at different locations in the cerebral cortex. In healthy patients, the blood oxygen saturation of the cerebral cortex is usually uniform across the entire surface of the brain. Therefore, a decrease in oxygenated hemoglobin measured in a specific area of ​​the cerebral cortex indicates that there may be a problem. Because the brain is a physically symmetrical organ, NIRS comparison measurements are usually made at corresponding locations in the left and right lobes of the brain. Therefore, measurements can be made and compared at mirrored locations in the left and right lobes of the brain. For normal patients, the two measurements should be very close. However, if the two measurements differ greatly, such as by 20% or more, or the blood oxygen content at a specific location is reduced by 20% or more compared to other locations in the brain, it may indicate that the blood vessels supplying blood to this area of ​​the brain are blocked, resulting in insufficient oxygenated blood. In the implementation of the present invention, comparison measurement methods such as these can be used. Therefore, the blood oximetry measurements obtained at specific locations in the cerebral cortex are coupled to the image processor 30 of the ultrasound system, where the blood oximetry measurements are processed together with the ultrasound data to provide the clinician with combined ultrasound / NIRS information, as described in detail below.

[0028] The transducer arrays 10a and 10b of the ultrasound system can transmit ultrasound waves into the patient's brain from opposite sides of the head, but other locations, such as the suboccipital acoustic window at the front of the head or the back of the skull, can also be used or used interchangeably. Most patients' heads can provide suitable acoustic windows for transcranial ultrasound at the temporal bones around and above the ears on both sides of the head. In order to transmit and receive echoes through these acoustic windows, the transducer arrays must maintain good acoustic contact at these locations, which can be achieved by holding the transducer arrays on the head with a head-mounted device. For example, Figure 2 A head mount 62 is shown with two matrix array probes 66 mounted on a mannequin head 60. The head mount may have a snap-on deformable acoustic mount 64 that allows the sensor array to be manipulated through its compliant contact surface and aimed at the intracerebral artery while maintaining acoustic contact with the temporal window. Acoustic coupling to the target may be achieved by integrating a mating spherical surface on the probe handle that allows the probe to be rotated in the head mount 62 until it is firmly and tightly coupled to the patient's temporal window.

[0029] Figure 3 A human model is illustrated with a NIRS cap 70 that contains a wired LED emitter and photodiode sensor 72 for illuminating the brain and receiving reflected infrared light from the brain. The emitters and sensors are coupled to the channels of the NIRS system for amplification and detection. The locations of the emitters and sensors in three-dimensional space around the cap are known, so that a selected emitter and sensor pair can be activated and the region of the cerebral cortex to be measured can be determined (identified) by x, y, z coordinates. Thus, the NIRS cap 70 provides a common three-dimensional coordinate system framework in which measurement locations on the patient's head can be spatially determined and related to each other.

[0030] Figure 3 Also shown is a white circle 74, which marks one of the locations on opposite sides of the head where the ultrasound transducers 10a and 10b can be positioned to reach (access, enter) the skull acoustic window. Since the location of the acoustic window 74 is located at a known position of the cap and is related to the location of the transmitter and sensor 72, the location of the ultrasound transducer can be determined in the same spatial coordinate system as the transmitter and sensor location. Therefore, the NIRS cap 70 provides a common coordinate system for NIRS and ultrasound measurements and images.

[0031] Since a head-mounted mechanism (such as the head-mounted device 62) is required to mount and support the probe, a handheld ultrasound probe (such as the head-mounted device 62) with a NIRS cap 70 is used. Figure 2The probe 66 in the NIRS cap is usually awkward and cumbersome. For the NIRS cap, it is preferred to use an ultrasound transducer 10a, 10b with a small size (small volume) configuration that allows the ultrasound transducer 10a, 10b to be placed or attached at a known position 74 (and vice versa) within the NIRS cap and maintain good acoustic contact with the patient's head. Figure 4 A preferred ultrasound probe 10, 10b for implementing the present invention is illustrated, which includes a small-size two-dimensional matrix array 77 of transducer elements. The matrix ultrasound transducer is made into a patch for affixing to the patient's body with double-sided medical tape 75 so as to remain in the same position within the NIRS cap 70 during surgery. A suitable matrix array patch is described in U.S. Pat. No. 6,685,647 (Savord et al.), which uses a dematching layer for a small-size assembly. The matrix array is formed in the form of a standard piezoelectric acoustic stack (laminate) and is connected to a microbeamformer ASIC behind the array by a ball grid or equivalent interconnect. Figure 4 A plan view of matrix array probes 10a and 10b is shown. A flexible ribbon cable 79 couples the matrix array and microbeamformer to the ultrasound system. In use, the central matrix array region 77 of the probe is acoustically coupled to the area of ​​interest of the patient's body using ultrasound gel.

[0032] Figure 5 The volumetric image fields 102 and 104 scanned by the matrix array transducers 10a and 10b when acoustically coupled to the temple to scan the skull 100 are illustrated. The clinician can image the cranial blood vessels in these volumetric image fields and turn the pyramid image field to different directions to search for blood clots that block the cranial blood flow. At each position of the image fields 102 and 104, the clinician can find the location of blood flow obstruction in the real-time image on the display, and can also capture (freeze) the cranial blood vessel image or cranial blood vessel map. When the vascular map is acquired and statically saved, the image can be enhanced (such as composite, signal averaging) to improve the resolution or ratio of the image, and can be operated on the screen and carefully examined at different points and different viewing angles to accurately search for the location of vascular obstruction. In this way, the clinician can diagnose vascular stenosis. If the clinician does not find signs of blood flow path obstruction when examining the vascular map, the clinician can turn the image field to another area of ​​the cranium and examine the vascular map of another image field. Clinicians can use the Doppler data of the vascular map or the spectral Doppler function of the ultrasound system to measure the blood flow velocity at specific points in the cranial and cerebral blood vessels indicated by the cursors 110 and 112 in the image for comparative analysis with the NIRS results.

[0033] It can be seen that the origin of each volumetric ultrasound image field 102, 104 is located at a point on the matrix array 10a, 10b. Since each matrix array is positioned at a known position 74 of the NIRS cap, the three-dimensional coordinates of each point in the ultrasound image field are in the same spatial reference system as the NIRS transmitter and sensor. Therefore, ultrasound image data and NIRS image data can be spatially correlated with each other because they share a common spatial coordinate system in two-dimensional or three-dimensional imaging.

[0034] Figure 6 An ultrasound image 80 generated from ultrasound image data acquired by one of the transducers 10a and 10b is illustrated. Bright points in the Doppler window of the image show the location of cranial blood flow in the image field. Figure 7 An example display of NIRS image data is shown. For reference, the brighter NIRS image area 82 is Figure 7 84, which indicates that there may be abnormal blood flow in the cortex on the right side of the image. As shown in the following figure, the combined ultrasound / NIRS system of the present invention advantageously displays ultrasound and NIRS image data in a common reference frame.

[0035] Figure 8 The display effect produced by the multi-modal system of the present invention is shown. Before generating the image display, the image processor 30 acquires the ultrasound image (pixels or stereo pixels) and the geometric position of the NIRS sensor (x, y, z and direction data (if any)), and processes it through a registration algorithm. Since both NIRS and ultrasound data are acquired in a common coordinate system, they can be registered to spatial coordinates to link the data. At the top of the display is a head model template 90, which links the NIRS and ultrasound data. Such head templates can be obtained from open source code libraries, such as the Colins27 average brain model of the McConaughey Brain Imaging Center. Alternatively, if there is a CT or MRI image of the patient's head, it can also be used as a template. The circle around the brain model 90 marks the location of the NIRS sensor, such as sensor 2 is marked at 722. The dashed box 92 to the right of sensor 2 marks the location in the brain where sensor 2 acquires NIRS data and detects blood flow abnormalities (hypoxia). Below the head model template 90 is an ultrasound image 80, which is obtained from the brain area marked by the dashed box. Ultrasound image 80 can be associated with the dashed box 92 and the area of ​​sensor 2 because the NIRS sensor and ultrasound transducer and data are associated with the common coordinate system described above. The sensor circles filled with color around the head template (such as circle 722) indicate that there are both NIRS data and ultrasound data; empty circles indicate that one or both data are missing at that brain location.

[0036] Fig. 9 This conceptually illustrates how NIRS data and ultrasound data are combined in the system of the present invention to form a co-registered image. Using the known hardware configuration of the NIRS sensor 72 in the NIRS cap 70, a table of sensor positions corresponding to the model can be obtained, such as Fig. 9 (a). Fig. 9 As shown in (c), the (x, y, z) coordinate positions in the table represent the arrangement (alignment) of the sensors 72 in the NIRS cap 70. Fig. 9 As shown in the table in (b), the abstract positions of brain structures can be calculated from their geometric centers in the NIRS data. After determining the positions of the NIRS sensor and the brain structure in the coordinate system of the NIRS cap, the real-time dynamic ultrasound image can be aligned to this coordinate system based on the detected features found in the ultrasound data. This method can be used even if the position of the ultrasound transducer is not physically referenced to the NIRS cap, but a free-hand-held ultrasound scan is performed. Alignment can be performed using an iterative method, that is, searching for all the best matches between the two-dimensional structure in the ultrasound image and its three-dimensional position relative to the NIRS cap. Alternatively, a linear regression method can also be used to optimize the search process and provide the best positioning of the ultrasound imaging plane or volume relative to the NIRS sensor.

[0037] Fig.10 A hardware subsystem is described for use when three-dimensional NIRS data and ultrasound data have been (eg Figure 3 The two data types are combined when referenced to a common coordinate system by the NIRS cap in the three-dimensional memory 120. The three-dimensional set of NIRS data is stored in the three-dimensional memory 120 at locations addressed by their (x, y, z) coordinates. Similarly, the three-dimensional set of ultrasound data is stored in the memory 120 at locations addressed by their (x, y, z) coordinates. The data can be combined (e.g., summed) in the memory 120. The stored three-dimensional data is then rendered into a three-dimensional display by the volume rendering processor 34, and the operation of the volume rendering processor 34 is as described in the patent of Entrekin et al. referenced above. This is achieved by processing the data on virtual lines obtained by projecting the three-dimensional data, which virtual lines pass through the volume in the desired viewing direction. Both types of image data encountered along the virtual lines are processed by known techniques, such as minimum or maximum intensity projections. Through user control of the control panel 38, the operator can change or continuously change the rendering parameters and viewing direction, providing a dynamic parallax display that gives the impression of viewing a rotating or moving volume.

[0038] Stroke regions and vascular structures in NIRS and ultrasound data can be detected through advanced processing (e.g., processing performed by a neural network or deep learning software running on an image processor). First, the neural network is trained with NIRS images and ultrasound images containing clinically determined (identified) stroke regions and vascular structures. When training the images, the areas that need to be detected are already annotated. After appropriate training with known images, the neural network or deep learning software can be used to segment diagnostically relevant areas in both types of images.

[0039] Fig.11a and Fig.11b A method for multimodal diagnosis of suspicious areas of the body using ultrasound imaging and NIRS imaging according to the present invention is described. In step 132, ultrasound image data of a body area is acquired using one or more ultrasound probes 66. In step 134, NIRS data of the body area is acquired in the same coordinate system as the ultrasound data. As described above, this can be accomplished in the following manner, i.e., one or more ultrasound matrix array transducers 10a, 10b are fixed to a NIRS cap 70, and then the NIRS cap 70 is mounted on the subject's head to acquire ultrasound and NIRS data of the patient's head. Therefore, the NIRS cap provides the same coordinate reference for ultrasound data and NIRS data. In step 136, an image of the body area is generated using ultrasound data and NIRS data. In step 138, one or more suspicious pathological areas are detected in the ultrasound and NIRS images. Detection can be manually accomplished by a clinician in the following manner, i.e., the clinician can visually inspect the image details and circle or highlight the suspicious area in the image. As described above, an anatomical vascular map can be used for detection. The detection may also or alternatively be performed automatically as described above by a neural network or deep learning software that has been trained to detect suspicious lesions in anatomical images of the subject. Fig.10 The process shown in the figure combines the ultrasound and NIRS images into a three-dimensional data set. Alternatively, the ultrasound and NIRS images can be displayed separately, such as side by side or alternately. In step 142, a suspected pathology area is selected in the image data. This can be done manually by the clinician using a pointing device such as a mouse or trackball to click on the suspected pathology area in the image or using a pointing device to outline the suspected pathology area. Then, in step 144, one or both modes of suspected pathology images are displayed. Since the ultrasound and NIRS images are displayed in a common coordinate system, the coordinates of the selected image area can be used to delineate and select the image of the suspected pathology from the image data of the other mode. The suspicious pathology can be highlighted in one or both images for quick identification and diagnosis. In step 146, as shown in FIG. Fig. 9As shown, diagnostic data related to suspected pathology detected in the image is displayed to the user. The data can be coordinate data or diagnostic data obtained by one or both imaging modes, for example, oxygenated hemoglobin measurements produced by NIRS systems and Doppler flow rates produced by Doppler technology in ultrasound systems. In this way, image areas related to suspected pathology obtained from both imaging modes and diagnostic data related to the diagnosis produced by both modes can be provided to the clinician.

[0040] It should be noted that ultrasound systems and NIRS systems (particularly Figure 1 The component structure of the combined ultrasound / NIRS system in the embodiment of the present invention can be implemented by hardware, software or a combination thereof. The various embodiments of the system and / or components and controllers thereof, or components and controllers therein, can also be implemented as part of one or more computers or microprocessors. The computer or processor may include a computing device, an input device, a display device and an interface (e.g., an interface for accessing the Internet). The computer or processor may include a microprocessor. The microprocessor may be connected to a communication bus, for example, to access a PACS system or a data network to import training images. The computer or processor may also include a memory. A memory device (such as a three-dimensional memory 120) may include a random access memory (RAM) and / or a read-only memory (ROM). The computer or processor may also include a storage device, which may be a hard disk drive or a removable storage drive, such as a floppy disk drive, an optical disk drive, a solid-state drive, etc. The storage device may also be other similar devices for loading computer programs or other instructions into a computer or processor.

[0041] The terms "computer" or "module" or "processor" or "workstation" as used herein may include any processor- or microprocessor-based system, including systems using microcontrollers, reduced instruction set computers (RISC), ASICs, logic circuits, and any other circuits or processors capable of performing the functions described herein. The above examples are exemplary only and are not intended to limit the definition and / or meaning of these terms in any way.

[0042] The computer or processor executes an instruction set stored in one or more storage elements to process input data. The storage element may also store data or other information as desired or required. The storage element may be in the form of an information source or a physical storage element in a processing machine. The instruction set of the ultrasound system (including the instruction set for controlling the acquisition, processing and display of ultrasound images) may include various commands as described above to instruct the computer or processor as a processing machine to perform specific operations (such as the above-mentioned registration algorithm and other methods and processes of various embodiments of the present invention). For example, the image processor may use software instructions to register different types of image data. The instruction set may be in the form of a software program. The software may be in various forms such as system software or application software, and may be embodied as a tangible non-temporary computer-readable medium. Many ultrasound and NIRS system functions are typically calculated by or under the guidance of software routines. In addition, the software may be a collection of separate programs or modules, or a program module or part of a program module in a larger program. The software may also include modular programming in the form of object-oriented programming. The processing of input data by the processing machine may be a response to an operator instruction, a response to a previous processing result, or a response to a request from another processing machine.

[0043] Furthermore, the following claim limitations are not drafted in a "means-plus-function" format and are not intended to be interpreted pursuant to 35 U.S.C. 112(6) unless and until such claim limitations expressly employ the phrase "means" followed by a recitation of function without further structure.

Claims

1. A method for diagnosing a suspicious area of ​​a body using multiple imaging modalities, the method comprising: acquiring image data of a body region using a first imaging modality; acquiring image data of the body region using a second imaging modality, wherein the image data of both imaging modalities are associated with a common coordinate system; generating an image of the body region using image data from each modality; Detect suspicious pathological areas in images of both modalities; Merging the image data of the detected areas into a three-dimensional data set; Select areas of suspected pathology; Displaying images of selected areas of suspected pathology in one or two modes; and Displays diagnostic information related to suspected pathology.

2. The method according to claim 1, wherein: Acquiring image data of the body region using the first imaging modality further comprises acquiring ultrasound image slices; and Wherein, using the second imaging mode to acquire image data of the body region further includes acquiring near infrared spectroscopy (NIRS) image data of the subject's head.

3. The method according to claim 2, wherein: Displaying the images of the two modalities of the selected suspected pathology area also includes registering the images of the two modalities to the head template to indicate the anatomical relationship of the images.

4. The method according to claim 1, wherein: Acquiring image data of the body region using the first imaging modality further comprises acquiring ultrasound image slices; and Wherein, using the second imaging mode to acquire image data of the body region further includes acquiring MR or CT image data of the subject's head.

5. The method according to claim 2, wherein: Detecting the suspicious pathological area also includes detecting the suspicious pathological area by processing NIRS and ultrasound images.

6. The method according to claim 5, wherein: Detecting suspicious pathological areas also includes detecting suspicious pathological areas using neural networks or deep learning software.

7. The method according to claim 6, wherein: Detecting suspicious pathology areas using a neural network or deep learning software also includes extracting two modes of images of the suspicious pathology for simultaneous display.

8. The method according to claim 7, wherein: Displaying the extracted suspicious pathology images of the two modes also includes displaying the extracted suspicious pathology images side by side or in superposition.

9. The method according to claim 7, wherein: Displaying the extracted suspected pathology images of both modalities further includes displaying the extracted images in anatomical registration with the anatomical template.

10. The method according to claim 6, further comprising: Use suspicious pathology image data to train neural networks or deep learning software.

11. The method according to claim 1, wherein: Merging the detected regions into a three-dimensional dataset also includes rendering the image data from both modalities together into one volumetric image.

12. The method according to claim 2, wherein: Acquiring image data of the body region using the first imaging modality further comprises acquiring ultrasound image data of the brain; and Wherein, using the second imaging mode to acquire image data of the body region further includes acquiring NIRS image data of the brain.

13. The method according to claim 2, wherein: Acquiring NIRS image data of the subject's head further includes acquiring NIRS image data from a plurality of NIRS emitters and sensors having a known spatial relationship to the ultrasound transducer array.

14. The method of claim 13, further comprising registering the NIRS image data and the ultrasound image data to a common spatial coordinate system based on features detected in the ultrasound data.

15. The method of claim 4, further comprising registering the ultrasound image slices and the MR or CT image data to a head anatomical template.

16. A tangible, non-transitory computer-readable medium containing computer-executable instructions, which, when executed on a computer, cause the computer to implement the following method: acquiring image data of a body region using a first imaging modality; Acquiring image data of the body region using a second imaging modality, wherein: The image data of both imaging modes are related to a common coordinate system; generating an image of the body region using image data from each modality; Detect suspicious pathological areas in images of both modalities; Merging the image data of the detected areas into a three-dimensional data set; Select areas of suspected pathology; Display images of selected areas of suspected pathology in one or two modes; as well as Displays diagnostic information related to suspected pathology.

17. A multi-mode imaging system for diagnosing suspicious areas of a body, the multi-mode imaging system comprising: a first imaging mode configured to acquire image data of a body region; a second imaging mode configured to acquire image data of a body region, wherein the image data of both imaging modes are associated with a common coordinate system; and An image processor, the image processor being configured to: generating an image of the body region using image data from each modality; Detect suspicious pathological areas in images of both modalities; Merging the image data of the detected areas into a three-dimensional data set; Select areas of suspected pathology; Displaying images of selected areas of suspected pathology in one or two modes; and Displays diagnostic information related to suspected pathology.

18. The multi-mode imaging system of claim 17, wherein: The first imaging modality further includes a near infrared spectroscopy (NIRS) system (50) adapted to acquire NIRS signals from the brain and generate NIRS image data using the acquired NIRS signals; and The second imaging mode further includes an ultrasound imaging system, which is suitable for acquiring ultrasound signals from the brain and generating ultrasound image data using the acquired ultrasound signals.

19. The multi-mode imaging system of claim 18, wherein: Displaying one or both modal images of a selected region of suspected pathology also includes spatially registering the NIRS image data and the ultrasound image data to the head template.

20. The multi-modality imaging system of claim 18, wherein: The image processor is further adapted to utilize neural network or deep learning software to detect stroke areas and vascular structures in NIRS and ultrasound image data.

Images