A two-dimensional semi-convolutional neural network for predicting hematoma expansion in non-contrast head computed tomography images

Abstract

A method and apparatus include: receiving a three-dimensional (3D) non-contrast computed tomography (NCCT) image of a head including a hematoma. Using the 3D NCCT image of the head including the hematoma to generate a plurality of two-dimensional (2D) images of the head including the hematoma. Using a first 2D convolutional neural network (CNN), generating a plurality of 2D hematoma images based on the plurality of 2D images. Identifying a 3D region of interest (ROI) containing the hematoma based on the plurality of 2D hematoma images. Generating a plurality of 2D images corresponding to the ROI. Using a second CNN, determining a hematoma expansion (HE) prediction score based on the plurality of 2D images corresponding to the ROI. Providing a HE prediction score.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 670,133 filed in the U.S. Patent and Trademark Office on October 31, 2019, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Convolutional neural networks (CNNs) have been used in various medical imaging applications, including classification, segmentation, and registration. However, in most applications, CNNs are based on two-dimensional (2D) images because three-dimensional (3D) models generally require more computation, memory space, and training data. In addition, many medical imaging modalities generate inherent 3D images, such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), etc. In order to process 3D images, many 3D CNN architectures have been proposed. Summary of the invention

[0004] According to some possible implementations, a method includes: receiving a three-dimensional (3D) non-contrast computed tomography (NCCT) image of a head including a hematoma; using the 3D NCCT image of the head including the hematoma to generate a plurality of two-dimensional (2D) images of the head including the hematoma; using a first 2D convolutional neural network (CNN) to generate a plurality of 2D hematoma images based on the plurality of 2D images; identifying a 3D region of interest (ROI) containing the hematoma based on the plurality of 2D hematoma images; generating a plurality of 2D images corresponding to the ROI; using a second CNN to determine a hematoma expansion (HE) prediction score based on the plurality of 2D images corresponding to the ROI; and providing the HE prediction score.

[0005] According to some possible implementations, a device includes: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code, the program code including: a receiving code configured to cause at least one processor to receive a three-dimensional (3D) non-contrast computed tomography (NCCT) image of a head including a hematoma; a first generating code configured to cause at least one processor to use the 3D NCCT image of the head including the hematoma to generate a plurality of two-dimensional (2D) images of the head including the hematoma; a second generating code configured to cause at least one processor to use a first 2D convolutional neural network (CNN) to generate a plurality of 2D hematoma images based on the plurality of 2D images; an identifying code configured to cause at least one processor to identify a 3D region of interest (ROI) containing a hematoma based on the plurality of 2D hematoma images; a third generating code configured to cause at least one processor to generate a plurality of 2D images corresponding to the ROI; a determining code configured to cause at least one processor to use a second CNN to determine a hematoma expansion HE prediction score based on the plurality of 2D images corresponding to the ROI; and a providing code configured to cause at least one processor to provide the HE prediction score.

[0006] According to some possible implementations, a non-transitory computer-readable medium stores instructions, which include one or more instructions, which, when executed by one or more processors of a device, cause the one or more processors to: receive a three-dimensional (3D) non-contrast computed tomography (NCCT) image of a head including a hematoma; use the 3D NCCT image of the head including the hematoma to generate multiple two-dimensional (2D) images of the head including the hematoma; use a first 2D convolutional neural network (CNN) to generate multiple 2D hematoma images based on the multiple 2D images; identify a 3D region of interest (ROI) containing the hematoma based on the multiple 2D hematoma images; generate multiple 2D images corresponding to the ROI; use a second CNN to determine a hematoma expansion (HE) prediction score based on the multiple 2D images corresponding to the ROI; and provide a HE prediction score. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A and Figure 1B is a schematic diagram of an example implementation described herein;

[0008] Figure 2 is a diagram of an example environment in which the systems and / or methods described herein may be implemented;

[0009] Figure 3 yes Figure 2 diagrams of example components of one or more devices; and

[0010] Figure 4is a flow chart of an example process for using a 2.5-dimensional convolutional neural network to provide a hematoma expansion prediction score. DETAILED DESCRIPTION

[0011] Although the models mentioned above are based on 3D CNN, they still suffer from heavy computational burden and dimensionality. For example, due to the limited memory of graphics processing unit (GPU), the batch size used to train 3D U-Net or V-Net for segmentation tasks is usually set to "1". These limitations restrict the application of 3D CNN.

[0012] Convolutional neural networks (CNNs) have been widely used in medical imaging applications involving both two-dimensional (2D) and three-dimensional (3D) models, with 2D CNNs generally being more common than 3D CNNs due to computational intensity and training data availability.

[0013] However, many medical images are 3D images in nature, such as non-contrast computed tomography (NCCT) images used for intracranial hemorrhage (ICH) detection. The present disclosure provides a method for processing 3D NCCT images for ICH detection and hematoma expansion (HE) prediction using 2D CNN.

[0014] Specifically, the present invention applies 2D U-Net on each 2D slice of the NCCT image to segment the hematoma. Then, a fixed-size 3D subvolume (called "region of interest (ROI)") containing the hematoma and some surrounding environment is determined from the hematoma mask, and finally each slice in the ROI is used as a channel of the input of the 2D classification CNN to predict HE. In this way, the computational limitations of the 3D CNN are circumvented, and HE prediction accuracy substantially similar to the performance of radiologists is achieved.

[0015] U-Net is a 2D CNN model for image segmentation, especially for biomedical image segmentation. The present disclosure provides a technique for generating a hematoma membrane by segmenting hematoma using U-Net in NCCT images. The architecture of U-Net is shown in Figure 1A For example,

[0016] like Figure 1BAs shown, after generating the hematoma membrane of all slices, the bounding box containing the hematoma can be determined. The size of the bounding box depends on the volume of the hematoma, so a sub-volume of a fixed size called ROI can be identified from the bounding box, so that the centers of the bounding box and the sub-volume coincide, and the size of the ROI can contain almost all possible hematomas (e.g., 192×192×14 ROIs for 512×512×30 NCCT images).

[0017] Once the ROI is extracted from the whole volume, each slice in the ROI is used as a feature channel of the input to a classification CNN (e.g., NASNet model) of the 2D image to predict HE.

[0018] The present disclosure allows 3D images commonly used in medical applications to be processed by 2D CNN. In addition, the present disclosure improves efficiency by avoiding the intensive calculation of 3D CNN.

[0019] Figure 2 2 is a diagram of an example environment 200 in which the systems and / or methods described herein may be implemented. Figure 2 As shown, environment 200 may include user device 210, platform 220, and network 230. The devices of environment 200 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0020] The user device 210 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 220. For example, the user device 210 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some implementations, the user device 210 may receive information from the platform 220 and / or send information to the platform 220.

[0021] The platform 220 includes one or more devices capable of providing a hematoma expansion prediction score using a 2.5-dimensional convolutional neural network, as described elsewhere herein. In some implementations, the platform 220 may include a cloud server or a group of cloud servers. In some implementations, the platform 220 may be designed as a modular platform so that certain software components may be swapped in or out according to specific needs. Therefore, the platform 220 may be easily and / or quickly reconfigured for different purposes.

[0022] In some implementations, as shown, the platform 220 can be hosted in a cloud computing environment 222. It should be noted that although the implementations described herein describe the platform 220 as being hosted in a cloud computing environment 222, in some implementations, the platform 220 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0023] Cloud computing environment 222 includes an environment hosting platform 220. Cloud computing environment 222 can provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 210) to know the physical location and configuration of the systems and / or devices hosting platform 220. As shown, cloud computing environment 222 may include a set of computing resources 224 (this set of computing resources is collectively referred to as "computing resources 224", and a single computing resource is referred to as "computing resource 224").

[0024] The computing resources 224 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 224 may control the platform 220. Cloud resources may include computing instances running in the computing resources 224, storage devices provided in the computing resources 224, data transmission devices provided by the computing resources 224, etc. In some implementations, the computing resources 224 may communicate with other computing resources 224 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0025] Further Figure 2 As shown, the computing resources 224 include a set of cloud resources, such as one or more applications ("APP") 224-1, one or more virtual machines ("VM") 224-2, virtualized storage ("VS") 224-3, one or more hypervisors ("HYP") 224-4, etc.

[0026] The applications 224-1 include one or more software applications that can be provided to or accessed by the user device 210 and / or the sensor device 220. The applications 224-1 can eliminate the need to install and run software applications on the user device 210. For example, the applications 224-1 can include software associated with the platform 220 and / or any other software that can be provided through the cloud computing environment 222. In some implementations, one application 224-1 can send information to / receive information from one or more other applications 224-1 through the virtual machine 224-2.

[0027] Virtual machine 224-2 includes a software implementation of a machine (e.g., a computer) that runs a program, similar to a physical machine. Depending on the degree and purpose of the virtual machine 224-2 to any actual machine, virtual machine 224-2 can be a system virtual machine or a process virtual machine. A system virtual machine can provide a complete system platform that supports the operation of a complete operating system ("OS"). A process virtual machine can run a single program and can support a single process. In some implementations, virtual machine 224-2 can run on behalf of a user (e.g., user device 210) and can manage the infrastructure of cloud computing environment 222, such as data management, synchronization, or long-term data transfer.

[0028] Virtualized memory 224-3 includes one or more storage systems and / or one or more devices using virtualization technology within the storage system or device of computing resource 224. In some implementations, in the environment of the storage system, the type of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage, so that the storage system can be accessed without considering physical storage or heterogeneous structures. Separation allows administrators of storage systems to have flexibility in how administrators manage storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This can optimize storage usage, server consolidation and / or non-interference file migration performance.

[0029] Hypervisor 224-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer such as computing resource 224. Hypervisor 224-4 may present a virtual operating platform to the guest operating systems and may manage the operation of the guest operating systems. Multiple instances of each operating system may share virtualized hardware resources.

[0030] The network 230 includes one or more wired networks and / or wireless networks. For example, the network 230 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0031] Figure 2 The number and arrangement of devices and networks shown are provided as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different Figure 2 The devices and / or networks shown are arranged differently. Figure 2 Two or more of the devices shown may be implemented in a single device, or Figure 2 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 200 may perform one or more functions described as being performed by another set of devices of environment 200.

[0032] Figure 3 300 is a diagram of example components of device 300. Device 300 may correspond to user device 210 and / or platform 220. Figure 3 As shown, device 300 may include a bus 310 , a processor 320 , a memory 330 , a storage component 340 , an input component 350 , an output component 360 , and a communication interface 370 .

[0033] The bus 310 includes components that allow communication between components of the device 300. The processor 320 is implemented in hardware, firmware, or a combination of hardware and software. The processor 320 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 320 includes one or more processors that can be programmed to perform functions. The memory 330 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 320.

[0034] Storage component 340 stores information and / or software related to the operation and use of device 300. For example, storage component 340 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium, and a corresponding drive.

[0035] Input components 350 include components that allow device 300 to receive information, such as input through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). In addition or alternatively, input components 350 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 360 include components that provide output information from device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0036] The communication interface 370 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 300 to communicate with other devices, such as through a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 370 may allow the device 300 to receive information from another device and / or provide information to another device. For example, the communication interface 370 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0037] Device 300 may perform one or more processes described herein. Device 300 may perform these processes in response to processor 320 executing software instructions stored by a non-transitory computer-readable medium such as memory 330 and / or storage component 340. In this document, a computer-readable medium is defined as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0038] The software instructions may be read into the memory 330 and / or storage component 340 from another computer-readable medium or from another device through the communication interface 370. When executed, the software instructions stored in the memory 330 and / or storage component 340 may cause the processor 320 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0039] Figure 3 The number and arrangement of components shown are provided as examples. In practice, device 300 may include additional components, fewer components, different components, or different components. Figure 3 Additionally or alternatively, one set of components (eg, one or more components) of device 300 may perform one or more functions described as being performed by another set of components of device 300.

[0040] Figure 4 4 is a flow chart of an example process 400 for providing a hematoma expansion prediction score using a 2.5-dimensional convolutional neural network. In some implementations, the platform 220 may perform Figure 4 In some implementations, the process may be performed by another device or group of devices (e.g., user device 210) that is separate from or includes platform 220. Figure 4 One or more process blocks.

[0041] like Figure 4 As shown, process 400 may include receiving a three-dimensional (3D) non-contrast computed tomography (NCCT) image of a head including a hematoma (box 410).

[0042] Further Figure 4 As shown, process 400 may include using a 3D NCCT image of the head including a hematoma to generate multiple two-dimensional (2D) images of the head including the hematoma (box 420).

[0043] Further Figure 4As shown, process 400 may include generating a plurality of 2D hematoma images based on a plurality of 2D images using a first 2D convolutional neural network (CNN) (block 430).

[0044] Further Figure 4 As shown, process 400 may include identifying a 3D region of interest (ROI) containing a hematoma based on multiple 2D hematoma images (box 440).

[0045] Further Figure 4 As shown, process 400 may include generating a plurality of 2D images corresponding to the ROI (block 450 ).

[0046] Further Figure 4 As shown, process 400 may include determining a hematoma expansion (HE) prediction score based on a plurality of 2D images corresponding to the ROI using a second CNN (block 460).

[0047] Further Figure 4 As shown, process 400 may include providing a HE prediction score (block 470).

[0048] Although Figure 4 Example blocks of process 400 are shown, but in some implementations, process 400 may include additional blocks, fewer blocks, different blocks, or different blocks. Figure 4 Depicted blocks are blocks that are arranged differently. Additionally or alternatively, two or more blocks of process 400 may be performed in parallel.

[0049] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations may be made in light of the foregoing disclosure or may be acquired from practice of the implementations.

[0050] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0051] Obviously, the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0052] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may only be directly dependent on one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0053] Unless explicitly described per se, the elements, actions or instructions used herein should not be understood as being essential or necessary. In addition, as used herein, the articles "one" and "an" are intended to include one or more items, and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case where the intention is only one item, the term "one" or similar language is used. In addition, as used herein, the term "having", "having", "containing" or similar terms are intended to be open terms. In addition, the phrase "based on" means "based on at least in part", unless otherwise explicitly stated.

Claims

1. A method for predicting hematoma expansion in a non-contrast computed tomography image of the head, comprising: receiving a three-dimensional 3D non-contrast computed tomography (NCCT) image of the head including the hematoma; generating a plurality of two-dimensional 2D images of the head including the hematoma using the three-dimensional 3D non-contrast computed tomography NCCT image of the head including the hematoma; Using a first 2D convolutional neural network (CNN), generating a plurality of 2D hematoma images based on the plurality of two-dimensional 2D images; Based on the multiple 2D hematoma images, determining a bounding box containing the hematoma in the multiple 2D hematoma images; Identify a 3D region of interest (ROI) of fixed size containing the hematoma from the bounding box; generating a plurality of 2D images corresponding to the 3D region of interest ROI; Using each of the multiple 2D images of the 3D region of interest ROI as a feature channel of an input of a second CNN to determine a hematoma expansion HE prediction score, wherein the second CNN is a NASNet model; and A HE prediction score for hematoma expansion is provided.

2. The method according to claim 1, wherein: The first 2D convolutional neural network CNN is a U-Net model.

3. The method according to any one of claims 1 to 2, wherein: The method further comprises: Using the first 2D convolutional neural network (CNN), identifying a hematoma based on the plurality of two-dimensional (2D) images; and The multiple 2D hematoma images are generated, wherein the hematoma is segmented based on the multiple two-dimensional 2D images.

4. The method according to claim 3, further comprising: Identifying the volume of the hematoma based on identifying the hematoma; as well as The size of the 3D region of interest ROI is determined based on the volume of the hematoma.

5. The method according to any one of claims 1 to 2, wherein: The multiple two-dimensional 2D images of the head including the hematoma correspond to a complete image of the head.

6. The method according to claim 5, wherein: The multiple 2D hematoma images are smaller than the complete image of the head.

7. A device for predicting hematoma expansion in a non-contrast computed tomography image of the head, comprising: at least one memory configured to store program code; At least one processor is configured to read the program code and operate according to the instructions of the program code, wherein the program code includes: receiving code configured to cause the at least one processor to receive a three-dimensional 3D non-contrast computed tomography (NCCT) image of the head including a hematoma; a first generating code configured to cause the at least one processor to generate a plurality of two-dimensional (2D) images of the head including the hematoma using the three-dimensional (3D) non-contrast computed tomography (NCCT) image of the head including the hematoma; A second generating code configured to cause the at least one processor to generate a plurality of 2D hematoma images based on the plurality of two-dimensional 2D images using a first 2D convolutional neural network CNN; The identification code is configured to enable the at least one processor to determine, based on the multiple 2D hematoma images, a bounding box containing the hematoma in the multiple 2D hematoma images; and to identify a 3D region of interest ROI of a fixed size containing the hematoma from the bounding box; a third generating code configured to cause the at least one processor to generate a plurality of 2D images corresponding to the 3D region of interest ROI; Determine a code configured to use each of the multiple 2D images of the 3D region of interest ROI as a feature channel of an input of a second CNN to determine a hematoma expansion HE prediction score, wherein the second CNN is a NASNet model; and Code is provided, configured to cause the at least one processor to provide the hematoma expansion HE prediction score.

8. The device according to claim 7, wherein: The first 2D convolutional neural network CNN is a U-Net model.

9. The device according to any one of claims 7 to 8, wherein: The device further comprises: A second recognition code configured to cause the at least one processor to use the first 2D convolutional neural network CNN to recognize a hematoma based on the plurality of two-dimensional 2D images; and The fourth generating code is configured to cause the at least one processor to generate the multiple 2D hematoma images, wherein the hematoma is segmented based on the multiple two-dimensional 2D images.

10. The device according to claim 9, wherein: The second identification code is further configured to cause the at least one processor to identify a volume of the hematoma based on identifying the hematoma; as well as The device further includes: a second determination code configured to cause the at least one processor to determine the size of the 3D region of interest ROI based on the volume of the hematoma.

11. The apparatus according to any one of claims 7 to 8, wherein: The multiple two-dimensional 2D images of the head including the hematoma correspond to a complete image of the head.

12. The device according to claim 11, wherein The multiple 2D hematoma images are smaller than the complete image of the head.

13. A non-transitory computer-readable medium storing instructions, the instructions comprising one or more instructions which, when executed by one or more processors of a device, cause the one or more processors to perform the method of any one of claims 1 to 6.

Images