Lesion identification method and apparatus for neural image, and electronic device and storage medium

Abstract

The present disclosure relates to the field of image processing, and in particular to a lesion identification method and apparatus for a neural image, and an electronic device and a storage medium. The method comprises: determining a neural image set to be identified comprising a first structure image, a first lesion image and a first metabolic image; performing image pre-processing on the neural image set to obtain an object image set comprising a second structure image, a second lesion image and a second metabolic image; inputting the object image set into a trained three-dimensional convolutional neural network to obtain a lesion location of a target object; and performing lesion location labeling on the first structure image on the basis of the lesion location of the target object, so as to obtain and display a lesion location image. In the present disclosure, by pre-processing a multi-modal neural image and performing lesion detection on the multi-modal neural image by means of a neural network, a lesion location in the neural image can be accurately determined. In addition, in the present disclosure, an original neural image is directly labeled on the basis of an identification result, so that a user can visually view the lesion location.

Description

Neuroimaging lesion identification method, device, electronic device and storage medium Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a method, device, electronic device, and storage medium for identifying lesions in neuroimaging. Background Art

[0002] Epilepsy is a sudden, recurrent, and transient brain dysfunction caused by abnormal discharge activity of brain neurons. It is a common functional neurological disease worldwide. However, some epileptic lesions are hidden in imaging, and the extent of resection is difficult to determine, which is challenging in current clinical diagnosis and treatment. However, despite the rapid development of neuroimaging technology, some imaging changes are hidden or "negative", making them difficult to detect with conventional visual reading, and the rate of missed diagnosis is very high. Accurate positioning is the key to successful surgical diagnosis and treatment. With the rapid development of computer technology, the technology of using computer-assisted lesion identification has been widely used in recent years, but the related technology still has the problem of accurate identification results.

[0003] Summary of the Invention

[0004] In view of this, the present disclosure proposes a method, device, electronic device and storage medium for neuroimaging lesion identification, aiming to improve the accuracy of lesion identification results.

[0005] According to a first aspect of the present disclosure, a method for identifying lesions in neuroimaging is provided, the method comprising:

[0006] Determining a set of neural images to be identified, the set of neural images comprising a first structural image, a first lesion image, and a first metabolic image acquired from a target object;

[0007] performing image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image;

[0008] Inputting the object image set into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object;

[0009] The first structural image is annotated with the lesion position according to the lesion position of the target object to obtain and display a lesion position image.

[0010] In a possible implementation, performing image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image includes:

[0011] performing image segmentation on the first structural image to obtain at least one segmented image including tissue in the target object;

[0012] performing image position correction on the first structural image, the first lesion image, and the first metabolic image;

[0013] extracting the target object in the first structural image, the first lesion image, and the first metabolic image after the image position correction according to the segmented image, to obtain a second structural image, a second lesion image, and a second metabolic image;

[0014] An object image set is determined according to the second structural image, the second lesion image, and the second metabolic image.

[0015] In a possible implementation, the target object is a brain, and the segmented image includes a cerebrospinal fluid image, a gray matter image, and a white matter image.

[0016] In a possible implementation, performing image position correction on the first structural image, the first lesion image, and the first metabolic image includes:

[0017] The first structural image, the first lesion image and the first metabolic image are subjected to front joint correction, registration and density normalization processing.

[0018] In a possible implementation, extracting the target object from the first structural image, the first lesion image, and the first metabolic image after image position correction according to the segmented image to obtain a second structural image, a second lesion image, and a second metabolic image includes:

[0019] determining an image mask for representing a position of the target object based on at least one of the segmented images;

[0020] Target objects in the first structural image, the first lesion image, and the first metabolic image after image position correction are extracted respectively according to the image mask to obtain a second structural image, a second lesion image, and a second metabolic image.

[0021] In a possible implementation, the convolutional neural network includes a plurality of sequentially connected convolutional layers, fully connected layers, and activation layers, and each of the convolutional layers includes three convolutional channels corresponding to the second structural image, the second lesion image, and the second metabolic image, respectively;

[0022] Inputting the object image set into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object includes:

[0023] performing parallel convolution on the second structural image, the second lesion image, and the second metabolic image in sequence according to the multiple sequentially connected convolution layers to obtain corresponding first feature images, second feature images, and third feature images, respectively;

[0024] Inputting the first feature image, the second feature image, and the third feature image into the fully connected layer and the activation layer to obtain a probability value of each pixel position in the first feature image being a lesion position;

[0025] The lesion position of the target object is determined according to the probability value of each pixel position.

[0026] In a possible implementation, marking the lesion location on the first structural image according to the lesion location of the target object, obtaining and displaying a lesion location image, includes:

[0027] determining a lesion image according to the lesion location of the target object;

[0028] The lesion image is overlapped with the first structural image to mark the lesion position, and a lesion position image is obtained and displayed.

[0029] According to a second aspect of the present disclosure, a neuroimaging lesion identification device is provided, the device comprising:

[0030] a set determination module, configured to determine a set of neural images to be identified, wherein the set of neural images includes a first structural image, a first lesion image, and a first metabolic image acquired from a target object;

[0031] an image preprocessing module, configured to perform image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image;

[0032] A position recognition module, configured to input the object image set into a trained three-dimensional convolutional neural network to obtain the lesion location of the target object;

[0033] The position marking module is used to mark the lesion position on the first structural image according to the lesion position of the target object, obtain a lesion position image and display it.

[0034] In a possible implementation, the image preprocessing module is further configured to:

[0035] performing image segmentation on the first structural image to obtain at least one segmented image including tissue in the target object;

[0036] performing image position correction on the first structural image, the first lesion image, and the first metabolic image;

[0037] extracting the target object in the first structural image, the first lesion image, and the first metabolic image after the image position correction according to the segmented image, to obtain a second structural image, a second lesion image, and a second metabolic image;

[0038] An object image set is determined according to the second structural image, the second lesion image, and the second metabolic image.

[0039] In a possible implementation, the target object is a brain, and the segmented image includes a cerebrospinal fluid image, a gray matter image, and a white matter image.

[0040] In a possible implementation, the image preprocessing module is further configured to:

[0041] The first structural image, the first lesion image and the first metabolic image are subjected to front joint correction, registration and density normalization processing.

[0042] In a possible implementation, the image preprocessing module is further configured to:

[0043] determining an image mask for representing a position of the target object based on at least one of the segmented images;

[0044] Target objects in the first structural image, the first lesion image, and the first metabolic image after image position correction are extracted respectively according to the image mask to obtain a second structural image, a second lesion image, and a second metabolic image.

[0045] In a possible implementation, the convolutional neural network includes a plurality of sequentially connected convolutional layers, fully connected layers, and activation layers, and each of the convolutional layers includes three convolutional channels corresponding to the second structural image, the second lesion image, and the second metabolic image, respectively;

[0046] The location identification module is further configured to:

[0047] performing parallel convolution on the second structural image, the second lesion image, and the second metabolic image in sequence according to the multiple sequentially connected convolution layers to obtain corresponding first feature images, second feature images, and third feature images, respectively;

[0048] Inputting the first feature image, the second feature image, and the third feature image into the fully connected layer and the activation layer to obtain a probability value of each pixel position in the first feature image being a lesion position;

[0049] The lesion position of the target object is determined according to the probability value of each pixel position.

[0050] In a possible implementation, the location marking module is further configured to:

[0051] determining a lesion image according to the lesion location of the target object;

[0052] The lesion image is overlapped with the first structural image to mark the lesion position, and a lesion position image is obtained and displayed.

[0053] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0054] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0055] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0056] In one possible implementation, a neural image set to be identified, including a first structural image, a first lesion image, and a first metabolic image, is determined, and then the neural image set is subjected to image preprocessing to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image. The object image set is input into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object, and then the lesion location of the first structural image is annotated according to the lesion location of the target object to obtain and display a lesion location image. The present disclosure accurately determines the lesion location in the neural image by preprocessing the multimodal neural image and performing lesion detection on the multimodal neural image through a neural network. At the same time, the present disclosure directly annotates the original neural image according to the recognition result, and the user can intuitively view the lesion location.

[0057] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0059] FIG1 shows a flow chart of a method for identifying lesions in neuroimaging according to an embodiment of the present disclosure.

[0060] FIG2 shows a schematic structural diagram of a convolutional neural network according to an embodiment of the present disclosure.

[0061] FIG3 is a schematic diagram showing a method for identifying lesions using neuroimaging according to an embodiment of the present disclosure.

[0062] FIG4 shows a schematic diagram of a lesion location image according to an embodiment of the present disclosure.

[0063] FIG5 shows a schematic diagram of a neuroimaging lesion identification device according to an embodiment of the present disclosure.

[0064] FIG6 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure.

[0065] FIG7 shows a schematic diagram of another electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0067] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0068] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0069] The method for identifying lesions in neuroimaging of the embodiment of the present disclosure can be performed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the method for identifying lesions in neuroimaging of the embodiment of the present disclosure by calling computer-readable instructions stored in a memory through a processor.

[0070] Figure 1 shows a flow chart of a method for identifying lesions in neurological imaging according to an embodiment of the present disclosure. As shown in Figure 1 , the method for identifying lesions in neurological imaging according to an embodiment of the present disclosure may include the following steps S10-S40.

[0071] Step S10: Determine a set of neural images to be identified.

[0072] In one possible implementation, an electronic device determines a neuroimage set including neuroimages whose lesion locations need to be identified. The neuroimage set may include a first structural image, a first lesion image, and a first metabolic image acquired from a target object. The target object may be any human organ for which lesion location determination is required, such as the brain, heart, lungs, and liver. The electronic device may directly acquire images of the target object using a neuroimage acquisition device to obtain the neuroimage set, or may receive a neuroimage set acquired and transmitted by another device.

[0073] Optionally, the first structural image may be a T1WI IMPRAGE, which is used to focus on characterizing the anatomical structure of the target object. The first lesion image may be a T2WI FLAIR, which is used to focus on characterizing the lesion structure of the target object. The first metabolic image may be an FDG-PET, which is used to focus on characterizing the metabolic status of the target object. Therefore, different images in the neuroimaging set can characterize the structural characteristics of the target object from different dimensions and multimodally. At the same time, the first structural image, the first lesion image, and the first metabolic image are all three-dimensional images, which can characterize the three-dimensional structure of the target object in three-dimensional space.

[0074] Step S20: performing image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image.

[0075] In one possible implementation, after determining the neural image set, the electronic device performs image preprocessing on multiple images in the neural image set according to a preset image processing method to obtain the second structural image, the second lesion image, and the second metabolic image corresponding to the first structural image, the first lesion image, and the first metabolic image, respectively, to determine the object image set used to characterize the structural characteristics of the target object.

[0076] Optionally, the image preprocessing process is used to extract and position-correct target objects from multiple images in a neurological image set. The image preprocessing process performed by the electronic device may include performing image segmentation on the first structural image to obtain at least one segmented image including tissue in the target object. Image position correction is then performed on the first structural image, the first lesion image, and the first metabolic image. The target objects in the first structural image, the first lesion image, and the first metabolic image after image position correction are extracted based on the segmented images to obtain the second structural image, the second lesion image, and the second metabolic image. The object image set is determined based on the second structural image, the second lesion image, and the second metabolic image. The image position correction process is used to align the positions of target objects in multiple images so that the positions of target objects in different images are consistent. The segmented image is used to extract the target object to obtain the required information from multiple images and remove other useless information from the first structural image, the first lesion image, and the first metabolic image.

[0077] Furthermore, embodiments of the present disclosure may first determine the type of target object and then perform image segmentation based on at least one tissue type included in the target object. For example, in the case where the target object is the brain, the image segmentation process may be to extract the cerebrospinal fluid portion, gray matter portion, and white matter portion of the first structural image through image segmentation, so that the resulting segmented image includes a cerebrospinal fluid image, a gray matter image, and a white matter image. Optionally, the image segmentation process may be implemented based on a pre-set existing tissue probability map of the target object.

[0078] In one possible implementation, the process of image position correction by the electronic device may be to perform front joint correction, registration and density normalization on each image in the neuroimage set in sequence, that is, the first structural image, the first lesion image and the first metabolic image may be subjected to front joint correction, registration and density normalization. Among them, the front joint correction is used to translate and rotate each image in six directions based on the position of the origin in the first structural image, the first lesion image and the first metabolic image to adjust the target object included therein to the same spatial position, and the origin is the midpoint of the anterior joint and the posterior joint in the brain tissue. The image registration process is used to arrange the first structural image, the first lesion image and the first metabolic image in the same preset spatial coordinate system. The density normalization process is used to reduce the influence of the bias field and can be achieved by any non-parametric non-uniform density normalization function.

[0079] Optionally, the execution order of the image position correction process and the image segmentation process in the embodiments of the present disclosure can be pre-set. For example, image segmentation and image position correction can be performed simultaneously, or in a pre-set execution order. When image position correction and image segmentation are performed in a pre-set execution order, the image segmentation process can also be interspersed with multiple sub-steps of the image position correction process.

[0080] In the embodiment of the present disclosure, after obtaining at least one segmented image through image segmentation and completing the position correction of the first structural image, the first lesion image, and the first metabolic image, the electronic device extracts the target object in the first structural image, the first lesion image, and the first metabolic image based on the segmented image to obtain the second structural image, the second lesion image, and the second metabolic image. Optionally, the target object in the image can be extracted by establishing an image mask to remove useless information other than the target object. That is, the electronic device can first determine an image mask for characterizing the position of the target object based on at least one segmented image, and the image mask is used to characterize the position of the target object as a whole in the image, or the position of each tissue that needs to be retained in the target object. Then, the target objects in the first structural image, the first lesion image, and the first metabolic image after the image position correction are extracted respectively according to the image mask to obtain the second structural image, the second lesion image, and the second metabolic image, and then the object image set is determined based on the second structural image, the second lesion image, and the second metabolic image.

[0081] Step S30: input the object image set into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object.

[0082] In one possible implementation, after determining the object image set, the electronic device can input the object image set into a trained three-dimensional convolutional neural network, and perform multimodal feature extraction from three dimensions through the convolutional neural network to accurately determine the location of the lesion in the target object. The convolutional neural network is a three-dimensional neural network for extracting three-dimensional image features, which may include multiple sequentially connected convolutional layers, as well as fully connected layers and activation layers. Optionally, each convolutional layer in the convolutional neural network includes three convolution channels, which are respectively used to perform convolution processing on the input second structural image, second lesion image, and second metabolic image.

[0083] Optionally, after the object image set is input into the convolutional neural network, the second structural image, the second lesion image, and the second metabolic image can be first convolved in parallel according to a plurality of sequentially connected convolutional layers to obtain the corresponding first feature image, second feature image, and third feature image. The first feature image, the second feature image, and the third feature image are then input into the fully connected layer and the activation layer to obtain the probability value of each pixel position in the first feature image being the lesion position, and then the lesion position of the target object is determined based on the probability value of each pixel position. For example, a pre-set probability threshold can be used to determine whether each pixel position is the location of the lesion, that is, the pixel position with a probability value greater than the probability threshold can be determined as the location of the lesion, and the pixel position with a probability value less than the probability threshold can be determined as the location of the non-lesion.

[0084] Furthermore, each three-dimensional image input into the convolutional neural network is divided into cubes corresponding to multiple pixels, and multiple convolutional layers connected in sequence are used to perform continuous convolution through multiple filters to independently classify each pixel in the image to estimate whether its position is the lesion location. Each convolutional layer can include multiple feature channels, which are used to convolve each input cube (for example, the cube obtained by the second structural image segmentation, the cube obtained by the second lesion image segmentation, and the cube obtained by the second metabolic image segmentation are input into a feature channel respectively), extract the feature map and input it into the channel corresponding to the next convolutional layer for feature extraction again to obtain a feature image Where l is the number of the current convolutional layer, m is the number of neurons in the current convolutional layer, and b m is the preset bias, n is the image number corresponding to the channel (for example, the second structural image, the second lesion image, and the second metabolic image are numbered 1, 2, and 3 respectively), and f is a nonlinear function whose kernel is a learned hidden weight matrix Therefore, the connection between adjacent convolutional layers It can be viewed as a convolution object of a four-dimensional kernel on the concatenated channels. Convolution is performed to finally obtain the first feature image, the second feature image and the third feature image of the second structural image, the second lesion image and the second metabolic image, wherein the series channel includes the same channel in multiple adjacent convolution layers, C l-1 The number of channels included in each convolutional layer. The posterior probability is further calculated through the fully connected layer and activation function (SoftMax function) A soft segmentation map is formed having a probability value corresponding to each pixel position. is the cth category of channel L at pixel position x∈N 3 Activation, C L is the total number of categories of channel L.

[0085] The receptive field is the receptor of a neuron in the activation layer and is considered as the influence of the input pixel neighborhood on the neuron activation. The size increases with each subsequent convolutional layer. It is given by the following three-dimensional vector: (x,y,z) is used to represent the pixel position of the pixel in the cube, and represents the size of the kernel and the stride vector of the receptive field of the lth layer, τ l It is determined by the product of the kernel function’s step length. Optionally, in order to facilitate accurate segmentation, τ can be pre-set. l =(1, 1, 1). The neuron receptor in the activation layer corresponds to the pixel position in the middle of the input cube, which can be recorded as If the input size δ is specified in advance in , the size of the channel in the lth convolutional layer can be calculated as: To avoid overfitting, the convolution kernel used in the convolution layer in the embodiment of the present disclosure is a separate 3 3 kernel, which can perform convolution faster and contain fewer weights, which is beneficial for the segmentation of natural images. 3 Compared to the core, the smaller 3 3 The number of cores has been reduced by about 5 3 / 3 3 ≈4.6 times the computing power, which can also reduce the number of training parameters during model training.

[0086] Furthermore, in order to add local and larger background information to the three-dimensional convolutional neural network, the present application can downsample each input image, and then add a second path to perform convolution processing on the downsampled image. This multi-channel three-dimensional convolutional neural network can process multi-scale input images simultaneously. Among them, high-level features (such as position in the brain) and detailed local structural appearance features can be learned separately through each path. Since the normal convolution path and the downsampled convolution path are convolved separately, and then classified together according to the two convolution results, they are decoupled in the structure of the convolutional neural network, so arbitrarily large background information can be processed by simply adjusting the downsampling coefficient. The size of the path can be adjusted independently according to the computing power and task, which may require a relative adjustment of the number of filters used for downsampling. In order to maintain the ability of dense reasoning, the correlation of the activation space in the last convolution layer of the two paths should be guaranteed. In a network that only uses an odd number of kernel steps, the displacement of the first convolution kernel that inputs the receptive field at normal resolution is required to be The displacement of the convolution kernel of the input receptive field of the second down-sampling convolution path is L1 is the first convolution path, and L2 is the second convolution path. Therefore, the size of the convolution kernel in the second convolution path is required to be F Dis the downsampling coefficient. From the above formula, we can see that the input size of the second convolution path is Extraction can be performed centered on the same image position.

[0087] After obtaining the first, second, and third feature maps corresponding to each input image through multiple sequentially connected convolutional layers in a convolutional neural network, the three feature maps are input into a fully connected layer and an activation layer to predict whether each pixel position is a lesion location, thereby obtaining a probability value for each pixel position being a lesion location. The fully connected layer within the convolutional neural network of the present disclosure can be a CRF model (fully connected three-dimensional conditional random field), which can model arbitrarily large pixel neighborhoods and is computationally simple.

[0088] Furthermore, during the training process, the disclosed embodiment can divide each pixel position in each input image into cubes, and calculate the model loss function based on the annotation label corresponding to each cube and the prediction result to adjust the model. Optionally, the loss function used in the three-dimensional convolutional neural network training process can be: Where θ is the model parameter of the convolutional neural network, I s and c s are the s-th input cube and the true label of V pixel positions, respectively, and B is the total number of cubes. v is the corresponding position in the convolutional layer channel.

[0089] Figure 2 illustrates a schematic diagram of the structure of a convolutional neural network according to an embodiment of the present disclosure. As shown in Figure 2, the convolutional neural network includes multiple sequentially connected convolutional layers, as well as fully connected layers and activation layers. Each convolutional layer includes three convolution channels, one for performing convolution processing on the input second structural image, the other for the second lesion image, and the second metabolic image, respectively. After the object image set is input into the convolutional neural network, the multiple sequentially connected convolutional layers sequentially perform parallel convolutions on the second structural image, the second lesion image, and the second metabolic image, respectively, to generate the corresponding first, second, and third feature images. The first, second, and third feature images are then input into the fully connected layers and activation layers to obtain a probability value for each pixel in the first feature image being a lesion location. The lesion location of the target object is then determined based on the probability value for each pixel location. For example, pixel locations with a probability value greater than 0.5 can be determined as lesion locations, while pixel locations with a probability value less than 0.5 can be determined as non-lesion locations.

[0090] Step S40: marking the lesion position on the first structural image according to the lesion position of the target object, obtaining and displaying a lesion position image.

[0091] In one possible implementation, after the electronic device extracts features from different dimensions based on a trained three-dimensional convolutional neural network and identifies the location of a lesion in a target object, it can annotate the lesion location on a first structural image that focuses on displaying the structure of the target object based on the lesion location of the target object, thereby obtaining a lesion location image. The lesion image can be first determined based on the lesion location, and then the lesion image and the first structural image are overlapped to annotate the lesion location. Thus, the lesion location image can be obtained and displayed to the user, allowing the user to intuitively observe the lesion in the target object.

[0092] FIG3 is a schematic diagram of a method for identifying lesions of a neurological image according to an embodiment of the present disclosure. As shown in FIG3 , when the target object is the brain, the embodiment of the present disclosure collects the brain to obtain a set of neurological images including a first structural image, a first lesion image, and a first metabolic image. Then, the first structural image is segmented by image preprocessing to obtain a cerebrospinal fluid image, a gray matter image, and a white matter image, and the image position is corrected for the first structural image, the first lesion image, and the first metabolic image. The second structural image, the second lesion image, and the second metabolic image are determined based on the segmented image and the position-corrected image, and are input into a three-dimensional convolutional neural network for feature extraction from three dimensions to accurately predict the probability that each pixel position belongs to the lesion position, so as to obtain the lesion position and annotate the lesion position on the first structural image and display it to the user.

[0093] Figure 4 shows a schematic diagram of a lesion location image according to an embodiment of the present disclosure. As shown in Figure 4, the present embodiment utilizes three different types of neuroimaging to jointly determine lesion location from different dimensions, improving the accuracy of the determination results. Furthermore, the lesion location is directly annotated and displayed on the first structural image, allowing the user to intuitively observe the lesion in the target object.

[0094] Based on the above technical features, the present disclosure can pre-process multimodal neuroimaging, perform position registration on target objects in different neural influences, retain information related to the lesion location in the target object, and remove other irrelevant interference factors to improve the accuracy of the prediction results. At the same time, the convolutional neural network extracts the features of the multimodal neuroimaging for lesion detection, accurately determines the lesion location in the neuroimaging, and directly annotates the original neuroimaging according to the lesion location, allowing users to intuitively view the lesion location.

[0095] FIG5 is a schematic diagram of a neurological imaging lesion identification device according to an embodiment of the present disclosure. As shown in FIG5 , the neurological imaging lesion identification device according to an embodiment of the present disclosure may include:

[0096] A set determination module 50 is configured to determine a set of neural images to be identified, wherein the set of neural images includes a first structural image, a first lesion image, and a first metabolic image acquired from a target object;

[0097] An image preprocessing module 51 is configured to perform image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image;

[0098] A position recognition module 52 is configured to input the object image set into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object;

[0099] The position marking module 53 is configured to mark the lesion position on the first structural image according to the lesion position of the target object, obtain a lesion position image, and display the image.

[0100] In a possible implementation, the image preprocessing module 51 is further configured to:

[0101] performing image segmentation on the first structural image to obtain at least one segmented image including tissue in the target object;

[0102] performing image position correction on the first structural image, the first lesion image, and the first metabolic image;

[0103] extracting the target object in the first structural image, the first lesion image, and the first metabolic image after the image position correction according to the segmented image, to obtain a second structural image, a second lesion image, and a second metabolic image;

[0104] An object image set is determined according to the second structural image, the second lesion image, and the second metabolic image.

[0105] In a possible implementation, the target object is a brain, and the segmented image includes a cerebrospinal fluid image, a gray matter image, and a white matter image.

[0106] In a possible implementation, the image preprocessing module 51 is further configured to:

[0107] The first structural image, the first lesion image and the first metabolic image are subjected to front joint correction, registration and density normalization processing.

[0108] In a possible implementation, the image preprocessing module 51 is further configured to:

[0109] determining an image mask for representing a position of the target object based on at least one of the segmented images;

[0110] Target objects in the first structural image, the first lesion image, and the first metabolic image after image position correction are extracted respectively according to the image mask to obtain a second structural image, a second lesion image, and a second metabolic image.

[0111] In a possible implementation, the convolutional neural network includes a plurality of sequentially connected convolutional layers, fully connected layers, and activation layers, and each of the convolutional layers includes three convolutional channels corresponding to the second structural image, the second lesion image, and the second metabolic image, respectively;

[0112] The position identification module 52 is further configured to:

[0113] performing parallel convolution on the second structural image, the second lesion image, and the second metabolic image in sequence according to the multiple sequentially connected convolution layers to obtain corresponding first feature images, second feature images, and third feature images, respectively;

[0114] Inputting the first feature image, the second feature image, and the third feature image into the fully connected layer and the activation layer to obtain a probability value of each pixel position in the first feature image being a lesion position;

[0115] The lesion position of the target object is determined according to the probability value of each pixel position.

[0116] In a possible implementation, the location marking module 53 is further configured to:

[0117] determining a lesion image according to the lesion location of the target object;

[0118] The lesion image is overlapped with the first structural image to mark the lesion position, and a lesion position image is obtained and displayed.

[0119] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0120] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0121] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0122] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0123] 6 shows a schematic diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0124] 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 (I / O interface), a sensor component 814 , and a communication component 816 .

[0125] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0126] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0127] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0128] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0129] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0130] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0131] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0132] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0133] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0134] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0135] FIG7 shows a schematic diagram of another electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to FIG7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 can include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0136] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM, FreeBSD TM or similar.

[0137] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0138] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0139] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0140] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0141] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0142] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0143] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0144] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0145] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0146] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for identifying lesions in neurological imaging, characterized in that: The method comprises: Determining a set of neural images to be identified, wherein the set of neural images includes a first structural image, a first lesion image, and a first metabolic image acquired from a target object; Performing image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image; Inputting the object image set into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object; The first structural image is annotated with the lesion position according to the lesion position of the target object to obtain and display a lesion position image.

2. The method according to claim 1, characterized in that The performing image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image comprises: Performing image segmentation on the first structural image to obtain at least one segmented image including tissues in the target object; performing image position correction on the first structure image, the first lesion image, and the first metabolic image; Extracting the target object in the first structural image, the first lesion image, and the first metabolic image after the image position correction according to the segmented image, to obtain a second structural image, a second lesion image, and a second metabolic image; A set of object images is determined according to the second structure image, the second lesion image and the second metabolic image.

3. The method according to claim 2, characterized in that The target object is the brain, and the segmented image includes a cerebrospinal fluid image, a gray matter image, and a white matter image.

4. The method according to claim 2 or 3, characterized in that: The performing image position correction on the first structural image, the first lesion image, and the first metabolic image includes: The first structural image, the first lesion image and the first metabolic image are subjected to front joint correction, registration and density normalization.

5. The method according to claim 2, characterized in that: The step of extracting the target object in the first structural image, the first lesion image, and the first metabolic image after the image position correction according to the segmented image to obtain a second structural image, a second lesion image, and a second metabolic image includes: Determining an image mask for representing the position of the target object based on at least one of the segmented images; The target objects in the first structural image, the first lesion image and the first metabolic image after the image position correction are respectively extracted according to the image mask to obtain a second structural image, a second lesion image and a second metabolic image.

6. The method according to claim 1, characterized in that The convolutional neural network includes a plurality of sequentially connected convolutional layers, fully connected layers and activation layers, and each of the convolutional layers includes three convolutional channels corresponding to the second structural image, the second lesion image and the second metabolic image respectively; The step of inputting the object image set into the trained three-dimensional convolutional neural network to obtain the lesion location of the target object includes: performing parallel convolution on the second structural image, the second lesion image, and the second metabolic image in sequence according to the plurality of sequentially connected convolutional layers to obtain a first feature image, a second feature image, and a third feature image corresponding to each other; Inputting the first feature image, the second feature image, and the third feature image into the fully connected layer and the activation layer to obtain a probability value of each pixel position in the first feature image being a lesion position; The lesion position of the target object is determined according to the probability value of each pixel position.

7. The method according to claim 1, characterized in that The step of marking the lesion position of the first structural image according to the lesion position of the target object to obtain and display a lesion position image includes: Determining a lesion image according to the lesion position of the target object; The lesion image is overlapped with the first structural image to mark the lesion position, and a lesion position image is obtained and displayed.

8. A neuroimaging lesion identification device, characterized in that: The device comprises: A set determination module, used to determine a set of neural images to be identified, wherein the neural image set includes a first structural image, a first lesion image, and a first metabolic image acquired from a target object; An image preprocessing module, used for performing image preprocessing on the neural image set to obtain an object image set including a second structural image, a second lesion image, and a second metabolic image; A position recognition module, used for inputting the object image set into the trained three-dimensional convolutional neural network to obtain the lesion position of the target object; The position marking module is used to mark the lesion position of the first structural image according to the lesion position of the target object, obtain the lesion position image and display it.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images