Medical Image Analysis Method and Device, Computer Equipment, and Storage Medium

Abstract

The present disclosure relates to a medical image analysis method and apparatus, a computer device, and a computer-readable storage medium. The medical image analysis method includes: receiving a medical image acquired by a medical imaging device; performing navigation processing on the medical image based on an analysis requirement to obtain a navigation trajectory, where the analysis requirement includes a disease to be analyzed; extracting a set of image patches along the navigation trajectory; extracting image features based on the set of image patches using a first learning network; and feeding the extracted image features to a second learning network, where the second learning network is configured to determine an analysis result based on the image features and the navigation trajectory. The present disclosure unifies the model framework by means of the navigation design, and can implement more accurate and targeted analysis of medical images based on the navigation trajectory in combination with the first learning network and the second learning network. Therefore, the complexity of medical image processing is reduced, the efficiency and accuracy of medical image analysis are improved, and the labor cost is reduced.

Description

[0001] This application is a divisional application of a Chinese invention patent application with the application number 202110299397.9, the application date of March 22, 2021, and the invention title of "Medical Image Analysis Method and Device, Computer Equipment and Storage Medium". Technical Field

[0002] The present disclosure relates to the technical field of medical image processing, and more particularly, to a medical image analysis method and device, computer equipment, and computer-readable storage medium. Background Art

[0003] Medical images can quickly obtain structural and functional images of internal tissues of the human body non-invasively or minimally invasively, so as to clearly understand the physiological structures and metabolic conditions of various human organs. At present, medical image analysis has become an essential clinical diagnosis assistance means. Through image processing technology and computer technology, this information can be effectively processed for auxiliary diagnosis, surgical planning, etc., with significant social benefits and broad application prospects.

[0004] In recent years, deep learning (DL), especially convolutional neural network (CNN), has rapidly become a research hotspot in medical image analysis. It can automatically extract implicit disease diagnosis features from large medical image data, and thus has gradually become an important technology in image screening and classification.

[0005] However, the existing technology can only detect and analyze lesions of a single disease type using a single model in medical image lesion detection and analysis. For example, a breast cancer classification model is used to detect and analyze breast cancer, a nuclear cataract grading model is used to detect and analyze nuclear cataract, or an Alzheimer's disease (AD) and mild cognitive impairment (MCI) classification model is used to detect and analyze Alzheimer's disease and mild cognitive impairment, etc. In clinical applications, it often occurs that the analysis of multiple different types of diseases is required. However, the existing technology can only establish multiple models for analysis of multiple different types of diseases. Therefore, when analyzing medical images, manual adjustment of the model settings is required for different diseases, resulting in low efficiency and high labor costs in medical image analysis. In addition, the existing technology can only analyze the entire medical image and cannot analyze the disease area of interest, resulting in high complexity of medical image processing, low efficiency and accuracy of medical image analysis, and unable to meet the requirements of clinical applications for accurate disease analysis and diagnosis. Summary of the Invention

[0006] The present disclosure is provided to solve the above problems existing in the prior art.

[0007] The present disclosure relates to a medical image analysis method, an apparatus, a computer device, and a computer-readable storage medium. The medical image analysis method performs navigation processing on a received medical image based on analysis requirements to obtain a navigation trajectory, and determines an analysis result by using a first learning network and a second learning network based on a set of image patches extracted along the navigation trajectory, which can perform more accurate and targeted analysis on medical images. Therefore, the complexity of medical image processing is reduced, the efficiency and accuracy of medical image analysis are improved, and the labor cost is reduced.

[0008] According to a first aspect of the present disclosure, there is provided a medical image analysis method. The medical image analysis method includes: receiving a medical image acquired by a medical imaging device; performing navigation processing on the medical image based on analysis requirements to obtain a navigation trajectory, where the analysis requirements include the disease to be analyzed; extracting a set of image patches along the navigation trajectory; extracting image features based on the set of image patches by using a first learning network; and feeding the extracted image features to a second learning network, where the second learning network is configured to determine an analysis result based on the image features and the navigation trajectory.

[0009] According to a second aspect of the present disclosure, there is provided a medical image analysis apparatus. The medical image analysis apparatus includes: a first receiving module configured to receive a medical image acquired by a medical imaging device; a navigation processing module configured to perform navigation processing on the medical image based on analysis requirements to obtain a navigation trajectory, where the analysis requirements include the disease to be analyzed; a first extraction module configured to extract a set of image patches along the navigation trajectory; a second extraction module configured to extract image features based on the set of image patches by using a first learning network; and a first feeding module configured to feed the extracted image features to a second learning network, where the second learning network is configured to determine an analysis result based on the image features and the navigation trajectory.

[0010] According to a third aspect of the present disclosure, there is provided a computer device including a memory and a processor. The memory is used to store one or more computer program instructions, and one or more computer program instructions are executed by the processor to implement a medical image analysis method.

[0011] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium storing computer program instructions, where the computer program instructions implement a medical image analysis method when executed by a processor.

[0012] Using a medical image analysis method, apparatus, computer device, and computer-readable storage medium according to various embodiments of the present disclosure, a model framework is unified using navigation design. This model framework can meet different requirements for different diseases, enabling a user to obtain a corresponding navigation trajectory simply by inputting an analysis requirement (type of disease), avoiding complex operations of changing different model settings according to different requirements, and reducing labor costs. Further, this method can collect more useful information at the cost of relatively small amounts of information by adopting a navigation trajectory. That is, under a unified framework, relatively rich and effective information of image patches is extracted as much as possible for various different analysis requirements. Additionally, this method determines an analysis result by using a first learning network and a second learning network based on a set of image patches extracted along the navigation trajectory, can quickly and accurately obtain a medical image related to the analysis requirement, and perform more precise and targeted analysis on the obtained medical image. Therefore, the complexity of medical image processing is reduced, and the efficiency and accuracy of medical image analysis are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the drawings that are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. The same reference numerals with alphabetic suffixes or different alphabetic suffixes may represent different instances of similar components. The drawings generally illustrate various embodiments by way of example and not limitation, and are used in conjunction with the description and the claims to explain the disclosed embodiments. When appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be an exhaustive or exclusive embodiment of the apparatus or method.

[0014] Figure 1 A flowchart showing a medical image analysis method according to an embodiment of the present disclosure;

[0015] Figures 2(a) to 2(f) A schematic diagram showing a navigation trajectory according to an embodiment of the present disclosure;

[0016] Figure 3 A schematic diagram showing a set of extracted image patches according to an embodiment of the present disclosure;

[0017] Figure 4 A flowchart showing a medical image analysis method according to an embodiment of the present disclosure;

[0018] Figure 5 A block diagram showing a medical image analysis apparatus according to an embodiment of the present disclosure;

[0019] Figure 6 A block diagram showing a medical image analysis apparatus according to an embodiment of the present disclosure; and

[0020] Figure 7A structural block diagram of a computer device according to an embodiment of the present disclosure is shown. Detailed implementation manners

[0021] To enable those skilled in the art to better understand the technical solutions of the present disclosure, the present disclosure will be described in detail below with reference to the accompanying drawings and specific implementation manners. The embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings and specific examples, but this is not a limitation to the present disclosure. For the various steps described herein, if there is no necessity for a sequential relationship between them, the order in which they are described as examples herein should not be regarded as a limitation. Those skilled in the art should know that they can be adjusted in order as long as the logic between them is not destroyed and the entire process cannot be implemented.

[0022] In addition, those of ordinary skill in the art should understand that the accompanying drawings provided herein are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0023] Unless the context clearly requires otherwise, the words "including", "comprising" and similar words throughout the specification and claims should be interpreted as having an inclusive meaning rather than an exclusive or exhaustive meaning; that is, the meaning of "including but not limited to".

[0024] In the description of the present disclosure, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0025] A medical image analysis method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 A flowchart of a medical image analysis method according to an embodiment of the present disclosure is shown. As Figure 1 shown, the medical image analysis method may include the step of receiving a medical image acquired by a medical imaging device (step 102).

[0027] Specifically, the computer device receives medical images acquired by a medical imaging device. Here, the computer device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., and the embodiments of the present disclosure do not limit this. Further, the computer device can obtain medical images from the medical imaging device in real time, or can obtain the medical images acquired by the medical imaging device from a server, and the embodiments of the present disclosure do not limit this. Here, the server can be an independent physical server, such as a Picture Archiving and Communication Systems (PACS), or can be a server cluster or a distributed system composed of multiple physical servers, or can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The computer device and the server can be directly or indirectly connected through wired or wireless communication methods, and the embodiments of the present disclosure do not limit this.

[0028] The medical imaging device can include an X-ray imaging device, a Magnetic Resonance Imaging (MRI) device, an ultrasonic imaging device, a nuclear medicine imaging device, a thermal imaging device, a medical optical imaging device, etc., and the embodiments of the present disclosure do not limit this. The medical image is an image acquired by a medical imaging device. The medical image can include X-ray images (for example, Computed Tomography (CT) images), MRI images, ultrasonic images, radionuclide images, etc., and the embodiments of the present disclosure do not limit this. The medical image can be a two-dimensional (2D) or three-dimensional (3D) image, and the 3D image can contain multiple 2D image slices. Further, the medical image can be an image of the liver and kidney, or can be a chest image, or can also be an image of other parts, and the embodiments of the present disclosure do not limit this.

[0029] In some embodiments, the medical image analysis method may further include a step of performing navigation processing on the medical image based on an analysis requirement to obtain a navigation trajectory (step 104), where the analysis requirement includes the disease to be analyzed.

[0030] Specifically, after receiving a medical image, the computer device performs navigation processing on the received medical image based on the analysis requirements to obtain a navigation trajectory. Here, the analysis requirements may include the disease to be analyzed, such as brain tumor, breast cancer, etc.; or may include the target part to be analyzed, such as coronary blood vessels, carotid arteries in the head and neck, etc.; or may also include navigation manual / automatic settings, such as one or more demarcation key points of the liver and kidneys obtained by manual annotation / automatic detection. In some embodiments, the analysis requirements may not only include the disease to be analyzed, but may also include analysis requirements customized by the user (physician) according to different application scenarios or needs, and the embodiments of the present disclosure do not limit this.

[0031] The navigation processing may be a classification learning network established based on the training process of deep learning. For different analysis requirements (different disease types), the navigation trajectory may be in different forms. Specifically, the navigation trajectory may be directed and may include the path or pattern of key points, or may include the corresponding template of the structure or functional area, where each structure or functional area in the corresponding template is associated with each other for the analysis requirements. Further, the navigation trajectory may be a local area path, a one-way path, a complete path, a closed-loop path, a tree structure, a graph structure, etc., and the embodiments of the present disclosure do not limit this.

[0032] Exemplarily, for the grading problem of fatty liver, the navigation trajectory may be the pattern formed by one or more demarcation key points of the liver and kidneys obtained by manual annotation / automatic detection, as shown in FIG. 2(a); or may be the path formed by one or more demarcation key points of the liver and kidneys obtained by manual annotation, as shown in FIG. 2(b). For vascular lesion detection, the navigation trajectory may be the vascular segment trend formed by the key points obtained by manual annotation / automatic detection, as shown in FIG. 2(c); or may be the complete or partial vascular centerline calculated automatically, as shown in FIG. 2(d). For head and neck detection, the navigation trajectory may be the point cloud map of the carotid arteries in the head and neck obtained by manual drawing / automatic detection, as shown in FIG. 2(e).

[0033] Exemplarily, for brain ischemia detection, the navigation trajectory may be the corresponding template of the corresponding structures or functional areas of the left and right brains obtained by manual drawing / automatic detection / registration. As shown in FIG. 2(f), the upper left and upper right figures are the original brain image of a sagittal section of the three-dimensional brain map and the image after structural partitioning, respectively; the lower left and lower right figures are the original brain image of another sagittal section of the three-dimensional brain map and the image after structural partitioning, respectively.

[0034] For example, as shown in the figure at the upper right corner of FIG. 2(f), the side close to "L" is the left brain, and the side close to "R" is the right brain. The left and right brain regions with the same name are the corresponding structurally associated regions. It can be seen from the figure that the left and right brains respectively include regions M1, M2, M3, C, L, I, and IC, that is, left brain region M1 corresponds to right brain region M1, left brain region M2 corresponds to right brain region M2, left brain region M3 corresponds to right brain region M3, left brain region C corresponds to right brain region C, and so on. Another example is as shown in the figure at the lower right corner of FIG. 2(f), where the left and right brains respectively include regions M4, M5, and M6, that is, left brain region M4 corresponds to right brain region M4, left brain region M5 corresponds to right brain region M5, and left brain region M6 corresponds to right brain region M6. When training the network, these pairs of structurally associated regions can be input together for classification training and prediction of ischemia. It should be noted that the data for training the network is correlated, that is, left brain region M1 will not be associated and classified with left brain region M2, nor will left brain region M1 be associated and classified with right brain region M2.

[0035] The inventors of the present application have confirmed that by inputting pairs of structure / function associated regions as a set, the model only needs to analyze the associated regions, which can improve the analysis efficiency and obtain more accurate analysis results at the same time.

[0036] In some embodiments, the medical image analysis method may further include a step of extracting a set of image patches along a navigation trajectory (step 106).

[0037] Specifically, the computer device can input the medical image into a preset segmentation network and perform image segmentation on the medical image along the navigation trajectory to obtain a set of image patches. Here, the segmentation network for segmenting the medical image can be a machine learning network such as a multi-level learning network and can be trained using supervised learning. The architecture of the segmentation network may include a stack of different blocks and layers, and each block and layer converts more than one input into more than one output. Examples of different layers may include more than one convolutional layer or fully convolutional layer, non-linear operator layer, pooling layer or subsampling layer, fully connected layer, and / or final loss layer, and each layer can be connected to an upstream layer and a downstream layer. Further, the segmentation network can be a traditional network such as V-Net or U-Net, or can also be a Pyramid Scene Parsing Network (PSPNet), or can also be a DeepLabV3 network, and the embodiments of the present disclosure do not limit this.

[0038] Image segmentation refers to dividing an image into several non - overlapping regions according to features such as grayscale, color, spatial texture, geometric shape, etc., so that these features show consistency or similarity within the same region, while showing obvious differences between different regions. Image segmentation methods based on deep learning can include, but are not limited to, image segmentation methods based on feature encoding, image segmentation methods based on region selection, image segmentation methods based on Recurrent Neural Network (RNN), image segmentation methods based on upsampling / transposed convolution, image segmentation methods based on improving feature resolution, image segmentation methods based on feature enhancement, image segmentation methods using Conditional Random Field (CRF) / Marcov Random Field (MRF), etc.

[0039] Since the navigation trajectory can include the path or pattern of key points or the corresponding templates of structures or functional areas, and each structure or functional area in the corresponding template is related to each other for analysis requirements, the image blocks in the set of image blocks to be extracted may be functionally or spatially related to each other. Thus, the way of extracting the set of image blocks along the navigation trajectory can reduce the amount of information of the image blocks to be analyzed compared with the ordinary way of extracting the set of image blocks. That is, using the navigation trajectory is equivalent to collecting more useful information at the cost of a relatively small amount of information.

[0040] Exemplarily, take the input medical image as a 2D liver - kidney ultrasound image. As Figure 3 shown, after receiving the 2D liver - kidney ultrasound image collected by the ultrasound imaging device, the computer device performs navigation processing on the 2D liver - kidney ultrasound image to obtain a liver - kidney ultrasound image with a navigation trajectory (i.e., the two key points "★" shown in Figure 2); further, the computer device intercepts the 2D liver - kidney ultrasound image with the two key points "★" centered on these two key points "★" to obtain two 2D image blocks, and the collective formed by summarizing these two 2D image blocks is the set of image blocks. When determining whether there is fatty liver and its classification, the liver - kidney brightness ratio is an important indicator. Therefore, using the key points or paths of the boundary as the navigation trajectory and intercepting the image blocks to be analyzed according to the navigation trajectory can include the brightness information of both the liver and the kidney, making the analysis result more reasonable and accurate.

[0041] In some embodiments, the medical image analysis method may further include the step of extracting image features using a first learning network based on the set of image blocks (step 108).

[0042] Specifically, the first learning network may include a convolutional neural network. A convolutional neural network is a type of feedforward neural network (FNN) that contains convolutional computations and has a deep structure, and is one of the representative algorithms of deep learning. A convolutional neural network has the ability of representative learning and can perform translation-invariant classification on input information according to its hierarchical structure. Image features may include color features, texture features, shape features, and spatial relationship features. Further, the image features may be a sequence of features of each image patch in the image patch set.

[0043] Feature extraction is a concept in computer vision and image processing, which refers to using a computer to extract image information and determine whether each point in an image belongs to an image feature. The result of feature extraction is to divide the points on the image into different subsets, and these subsets often belong to isolated points, continuous curves, or continuous regions. The methods of feature extraction may include, but are not limited to, Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradient (HOG), Speeded Up Robust Feature (SURF), Difference of Gaussian (DOG), etc.

[0044] In some embodiments, the medical image analysis method may further include the step of feeding the extracted image features to a second learning network (step 110), where the second learning network is configured to determine an analysis result based on the image features and the navigation trajectory.

[0045] Specifically, the computer device feeds the extracted image features to the second learning network, so that the second learning network can analyze the medical image based on the image features and the navigation trajectory to obtain an analysis result. Here, the second learning network may include one or a combination of a recurrent neural network and a recursive neural network (RNN). A recurrent neural network is a type of recursive neural network that takes sequential data as input, recurs in the evolution direction of the sequence, and all nodes (recurrent units) are connected in a chain. A recursive neural network is an artificial neural network (ANN) with a tree-like hierarchical structure and the network nodes recursively process the input information according to their connection order, and is one of the deep learning algorithms. In some embodiments, the information interaction manner between the nodes of the second learning network may follow the physical (including but not limited to space, physiological functions, etc.) constraint relationships between the points on the corresponding navigation trajectory, so as to obtain a more accurate analysis result.

[0046] Exemplarily, for the detection of brain tumors, a template based on the navigation of brain structures or functional areas can be used, and a 3D convolutional neural network can be used to make predictive analysis; for the detection of vascular diseases, a 2D / 3D convolutional neural network can be used, and at the same time, based on the navigation of the trajectory of key points of a certain section of blood vessels, combined with a recurrent neural network or a recursive neural network to make predictive analysis.

[0047] According to the medical image analysis method of the embodiments of the present disclosure, the design of its navigation unifies the model framework, which can meet the different requirements of different diseases, so that the user only needs to input the analysis requirements (the type of disease) to obtain the corresponding navigation trajectory, avoiding the complex operation of changing multiple model settings according to different requirements and reducing the labor cost. Further, this method can collect more useful information at the cost of relatively small information volume by adopting the navigation trajectory. That is to say, under the unified framework, relatively rich and effective image block information is extracted as much as possible for various different analysis requirements. In addition, this method determines the analysis result by using the first learning network and the second learning network based on the set of image blocks extracted along the navigation trajectory, can quickly and accurately obtain the medical images related to the analysis requirements, and perform more accurate and targeted analysis on the obtained medical images. Therefore, the complexity of medical image processing is reduced, and the efficiency and accuracy of medical image analysis are improved.

[0048] In some embodiments, the image feature is a sequence of the features of each image block in the set of image blocks; the second learning network is configured to: perform spatial / functional relationship learning based on the image block features extracted by the first learning network and the navigation trajectory, and determine the analysis result based on the spatial / functional relationship between the learned image block features.

[0049] Specifically, the spatial relationship refers to the mutual spatial position or relative direction relationship between multiple objects segmented in the image, and these relationships can be divided into connection / adjacency relationships, overlap / overlap relationships, inclusion / containment relationships, etc. There are two methods for extracting image spatial relationship features: one is to automatically segment the image, divide the objects or color regions contained in the image, then extract image features based on these regions, and establish an index; the other is to evenly divide the image into several regular sub-blocks, then extract features from each image sub-block, and establish an index. In some embodiments, the functional relationship is, for example, the functional correlation between the left and right brain regions, and so on.

[0050] In some embodiments, the medical image analysis method further includes: receiving the medical record of the subject of the medical image; screening out relevant texts required for analysis from the medical record; extracting text features based on the relevant texts; fusing the text features with image features to obtain fused features; and feeding the fused features into a second learning network, which is configured to determine an analysis result based on the fused features.

[0051] Specifically, the computer device screens out relevant texts required for analysis from the medical record of the subject of the received medical image, and extracts text features based on the relevant texts; further, the computer device performs feature fusion on the text features and the image features to obtain fused features, and feeds the fused features into the second learning network, so that the second learning network can determine an analysis result based on the fused features.

[0052] Here, the medical record is a written record made by medical staff about the patient's illness course and treatment situation. It is the basis for doctors to diagnose and treat diseases and valuable materials for medical scientific research. The medical record may include basic information (e.g., name, gender, age, marital status, etc.), chief complaint, current medical history (e.g., onset time, acute or chronic, location, characteristics and changes of accompanying symptoms, etc.), past medical history (e.g., past health status, whether having suffered from infectious diseases, whether having a surgical history, whether having a food or drug allergy history, etc.), personal history (e.g., whether life is regular, whether having bad habits, whether having a history of major mental trauma, etc.), family history (e.g., whether having a family genetic disease, whether having a family aggregation disease, whether there is a similar disease among family members, etc.). The criteria for case screening may include but are not limited to case inclusion and exclusion criteria, classification criteria, etiology grouping criteria, age grouping criteria, prognosis grouping criteria, etc. It should be noted that cases can be screened manually or automatically, and the embodiments of the present disclosure do not limit this.

[0053] Text feature extraction refers to extracting features from text that can effectively represent the overall information of the text. Text feature extraction can effectively reduce the dimension of the text vector space from high-dimensional to low-dimensional mapping, and can screen out feature items representing categories. The methods of text feature extraction may include but are not limited to transforming the original features into fewer new features by mapping or transformation methods, selecting some of the most representative features from the original features, selecting the most influential features according to experts' knowledge, and selecting by mathematical methods to find the features with the most classification information, etc.

[0054] In the embodiments of the present disclosure, text features can be extracted based on relevant texts and by using a Natural Language Processing (NLP) model. Here, natural language processing is a discipline that takes language as the object and uses computer technology to analyze, understand, and process natural language. That is, the computer is used as a powerful tool for language research, and quantitative research on language information is carried out with the support of the computer, and language descriptions that can be commonly used between humans and computers are provided. Natural language processing includes two parts: Natural Language Understanding (NLU) and Natural Language Generation (NLG). The natural language processing model can be a language model, an N-gram model, or a Neural Network Language Model (NNLM), and the embodiments of the present disclosure do not limit this.

[0055] Feature fusion refers to integrating features from different sources together and removing redundancy to obtain fusion features that are more conducive to predicting and analyzing medical images. The algorithms for feature fusion can include, but are not limited to, feature fusion algorithms based on Bayesian theory (such as the multiplication rule and addition rule of classifiers, feature fusion algorithms based on linear feature dependence models, etc.), feature fusion algorithms based on sparse representation theory, feature fusion algorithms based on deep learning theory, etc. Here, the feature fusion algorithm based on sparse representation theory is to establish a feature joint sparse matrix after extracting multiple features from the samples, and this feature joint sparse matrix is the result of multi-feature fusion; the feature fusion algorithm based on deep learning theory is to fuse the features obtained by multiple neural networks to obtain fusion features.

[0056] In some embodiments, further including fusing text features and image features to obtain fusion features: merging the text features and image features, and feeding the merged features into a multi-layer perceptron, and optimizing the merged features through backpropagation to obtain fusion features.

[0057] Specifically, the computer device merges the text features and image features, and inputs the merged features into a multi-layer perceptron; further, the computer device uses the backpropagation algorithm to optimize the merged features to obtain fusion features.

[0058] Here, feature merging refers to superimposing features from different sources together to obtain merged features that are more conducive to predictive analysis of medical images. The ways of feature merging can include the add method and the concat method. For network structures such as ResNet / FPN, where values are superimposed element by element and the number of channels remains unchanged, the add method can be used to fuse features; while for network structures such as DenseNet, channel merging is performed, and the concat method can be used to fuse features; that is to say, the concat method is to combine features, fuse the features extracted by multiple convolutional feature extraction frameworks, or fuse the information of the output layer; while the add method is the superimposition between information. Further, the add method increases the amount of information in each dimension of the features describing the image without changing the dimension, which is obviously beneficial to the final classification of the image; while the concat method is the merging of the number of channels, that is, the dimension of the features describing the image increases, and the amount of information in each dimension of the features remains unchanged.

[0059] The Multilayer Perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set. Further, the multilayer perceptron is a neural network architecture that can have multiple hidden layers in addition to the input layer and the output layer.

[0060] The Backpropagation (BP) algorithm is a learning algorithm suitable for multi-layer neuron networks, which is based on the gradient descent method. The main idea of the BP algorithm is to input the training set data into the input layer of the artificial neural network, pass through the hidden layer, and finally reach the output layer and output the result, which is the forward propagation process of the artificial neural network; since there is an error between the output result of the artificial neural network and the actual result, the error between the estimated value and the actual value is calculated, and this error is propagated backward from the output layer to the hidden layer until it reaches the input layer; during the backward propagation process, various parameter values are adjusted according to the error, and the above process is continuously iterated until convergence.

[0061] Figure 4 The flowchart showing the medical image analysis method according to an embodiment of the present disclosure is as follows Figure 4As shown, the medical image analysis method includes the following steps. The method starts with receiving a medical image acquired by a medical imaging device (step 402). The method may further include a step of performing navigation processing on the medical image based on analysis requirements to obtain a navigation trajectory, where the analysis requirements include the disease to be analyzed (step 404). The method may further include a step of extracting a set of image patches along the navigation trajectory (step 406). The method may further include a step of extracting image features based on the set of image patches using a first learning network (step 408). The method may further include a step of receiving the medical record of the subject of the medical image (step 410). The method may further include a step of screening relevant text of the analysis requirements from the medical record (step 412). The method may further include a step of extracting text features based on the relevant text (step 414). The method may further include a step of fusing the text features with the image features to obtain fused features (step 416). The method may further include a step of feeding the fused features into a second learning network (step 418), where the second learning network is configured to determine an analysis result based on the fused features.

[0062] According to the medical image analysis method of an embodiment of the present disclosure, by performing navigation processing on the received medical image based on analysis requirements to obtain a navigation trajectory, extracting image features using a first learning network based on a set of image patches extracted along the navigation trajectory, and fusing text features extracted from relevant text of the analysis requirements screened from the medical record with the extracted image features to obtain fused features, and determining an analysis result based on the fused features using a second learning network, it is possible to reduce the interference of redundant information, so that it is possible to quickly and accurately obtain medical images related to the analysis requirements and perform more accurate and targeted analysis on the obtained medical images. Therefore, the effectiveness of information is ensured, the complexity of medical image processing is reduced, the efficiency and accuracy of medical image analysis are improved, and the labor cost is reduced.

[0063] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiment of the present disclosure. For details not disclosed in the apparatus embodiment of the present disclosure, please refer to the method embodiment of the present disclosure.

[0064] Figure 5 A block diagram showing a medical image analysis apparatus according to an embodiment of the present disclosure is as follows Figure 5 As shown, the medical image analysis apparatus may include a first receiving module 502, which is configured to receive a medical image acquired by a medical imaging device.

[0065] In some embodiments, the medical image analysis apparatus may further include a navigation processing module 504, which is configured to perform navigation processing on the medical image based on analysis requirements to obtain a navigation trajectory, where the analysis requirements include the disease to be analyzed.

[0066] In some embodiments, the analysis requirements include not only the disease to be analyzed, but also, for example, analysis requirements customized by the user according to different application scenarios.

[0067] In some embodiments, the medical image analysis device may further include a first extraction module 506 configured to extract a set of image patches along a navigation trajectory. Since the navigation trajectory may include corresponding templates associated with structural or functional regions, the image patches in the set of image patches to be extracted may be functionally or spatially associated with each other.

[0068] In some embodiments, the medical image analysis device may further include a second extraction module 508 configured to extract image features based on the set of image patches using a first learning network.

[0069] In some embodiments, the medical image analysis device may further include a first feeding module 510 configured to feed the extracted image features to a second learning network, which is configured to determine an analysis result based on the image features and the navigation trajectory.

[0070] According to the medical image analysis device of the embodiments of the present disclosure, it unifies the model framework by using a navigation processing module. This model framework can meet the different requirements of different diseases, enabling the user to obtain a corresponding navigation trajectory by simply inputting the analysis requirements (type of disease), avoiding the complex operation of changing multiple model settings according to different requirements and reducing the labor cost. Further, the device uses the navigation trajectory to collect as much useful information as possible at the cost of relatively small information volume. That is, under the unified framework, relatively rich and effective information of image patches is extracted as much as possible for various different analysis requirements. In addition, the device determines the analysis result based on the set of image patches extracted along the navigation trajectory using a first learning network and a second learning network, can quickly and accurately obtain medical images related to the analysis requirements, and perform more accurate and targeted analysis on the obtained medical images. Therefore, the complexity of medical image processing is reduced, and the efficiency and accuracy of medical image analysis are improved.

[0071] In some embodiments, the navigation trajectory is directed, the image features are a sequence of features of each image patch in the set of image patches, the first learning network includes a convolutional neural network, and the second learning network includes one or a combination of a recurrent neural network and a recursive neural network.

[0072] In some embodiments, the analysis requirements further include the target part to be analyzed and the navigation manual / automatic setting.

[0073] In some embodiments, the navigation trajectory includes a path or pattern of key points, or a corresponding template of a structure or functional area, and each structure or functional area in the corresponding template is associated with each other for analysis requirements.

[0074] In some embodiments, for the problem of fatty liver grading, the navigation trajectory is a path or pattern formed by one or more demarcation key points of the manually labeled / automatically detected liver and kidney.

[0075] In some embodiments, for vascular lesion detection, the navigation trajectory is the trend of a vascular segment formed by key points obtained through manual labeling / automated detection, or a complete or partial vascular centerline calculated automatically.

[0076] In some embodiments, for cerebral ischemia detection, the navigation trajectory is a corresponding template of the structures or functional areas corresponding to the left and right brains obtained through manual delineation / automated detection.

[0077] In some embodiments, the image feature is a sequence of features of each image patch in the image patch set; the second learning network is configured to: learn the spatial / functional relationships between the image patch features extracted by the first learning network based on the navigation trajectory, and determine the analysis result based on the learned spatial / functional relationships between the image patch features.

[0078] In some embodiments, the navigation process is a classification learning network established based on a deep learning training process.

[0079] Figure 6 A block diagram showing a medical image analysis device according to an embodiment of the present disclosure. As Figure 6 shown, the medical image analysis device may include a first receiving module 602 configured to receive medical images acquired by a medical imaging device.

[0080] In some embodiments, the medical image analysis device may further include a navigation processing module 604 configured to perform navigation processing on the medical image based on analysis requirements to obtain a navigation trajectory, where the analysis requirements include the disease to be analyzed.

[0081] In some embodiments, the medical image analysis device may further include a first extraction module 606 configured to extract a set of image patches along the navigation trajectory.

[0082] In some embodiments, the medical image analysis device may further include a second extraction module 608 configured to extract image features using a first learning network based on the set of image patches.

[0083] In some embodiments, the medical image analysis device may further include a first feeding module 610 configured to feed the extracted image features to a second learning network, which is configured to determine an analysis result based on the image features and the navigation trajectory.

[0084] In some embodiments, the medical image analysis device may further include a second receiving module 612 configured to receive the medical record of the subject of the medical image.

[0085] In some embodiments, the medical image analysis device may further include a text screening module 614 configured to screen out relevant texts of the analysis requirements from the medical record.

[0086] In some embodiments, the medical image analysis device may further include a third extraction module 616 configured to extract text features based on the relevant texts.

[0087] In some embodiments, the medical image analysis device may further include a feature fusion module 618 configured to fuse the text features and the image features to obtain fused features.

[0088] In some embodiments, the medical image analysis device may further include a second feeding module 620 configured to feed the fused features to a second learning network, which is configured to determine an analysis result based on the fused features.

[0089] According to the medical image analysis device of the embodiments of the present disclosure, by performing navigation processing on the received medical image based on the analysis requirements to obtain a navigation trajectory, extracting image features by using a first learning network based on a set of image patches extracted along the navigation trajectory, and fusing the text features extracted from the relevant texts of the analysis requirements screened out from the medical record with the extracted image features to obtain fused features, and determining an analysis result based on the fused features by using a second learning network, it is possible to reduce the interference of redundant information, so that it is possible to quickly and accurately obtain medical images related to the analysis requirements, and perform more accurate and targeted analysis on the obtained medical images. Therefore, the effectiveness of information is ensured, the complexity of medical image processing is reduced, the efficiency and accuracy of medical image analysis are improved, and the labor cost is reduced.

[0090] In some embodiments, the text features are extracted based on the relevant texts by using a natural language processing model.

[0091] In some embodiments, the feature fusion module 618 is further configured to merge the text features and the image features, and feed the merged features into a multi-layer perceptron to optimize the merged features through backpropagation to obtain fused features.

[0092] In some embodiments, the navigation processing is a classification learning network established for a deep learning-based training process.

[0093] Figure 7 FIG. shows a structural block diagram of a computer device according to an embodiment of the present disclosure. As Figure 7 shown, the computer device is a general-purpose data processing device, including a general computer hardware structure. The computer device at least includes a processor 702 and a memory 704. The processor 702 and the memory 704 are connected by a bus 706. The memory 704 is adapted to store instructions or programs executable by the processor 702. The processor 702 can be an independent microprocessor or a collection of one or more microprocessors. Thus, by executing the commands stored in the memory 704, the processor 702 implements the processing of data and the control of other devices by executing the method flow of the embodiment of the present disclosure as described above. The bus 706 connects the above-mentioned multiple components together and at the same time connects the above-mentioned components to a display controller 708, a display device, and an input / output (I / O) device 710. The input / output (I / O) device 710 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a body sensing input device, a printer, and other devices well known in the art. Typically, the input / output (I / O) device 710 is connected to the system through an input / output (I / O) controller 712.

[0094] Among them, the memory 704 can store software components, such as an operating system, a communication module, an interaction module, and an application program. Each of the above-mentioned modules and application programs corresponds to a set of executable program instructions for completing one or more functions and the methods described in the embodiments of the invention.

[0095] In some embodiments, the computer device can be located somewhere, scattered in multiple places, or can be a distributed computer device, such as being set in the cloud. The embodiments of the present disclosure do not limit this.

[0096] The flowcharts and / or block diagrams of the methods, systems, and computer program products according to the embodiments of the present disclosure described above depict various aspects of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and the combination of blocks in the flowchart legend and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing devices to generate a machine, so that the instructions (executed by the processor of the computer or other programmable data processing devices) create a device for implementing the functions / actions specified in the blocks of the flowchart and / or block diagram.

[0097] Meanwhile, as those skilled in the art will realize, various aspects of the embodiments of the present disclosure can be implemented as a system, a method, or a computer program product. Accordingly, various aspects of the embodiments of the present disclosure can take the following forms: a full hardware implementation, a full software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software aspects and hardware aspects that can generally be referred to herein as "circuits", "modules", or "systems". In addition, aspects of the present disclosure can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0098] Any combination of one or more computer-readable media can be utilized. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example (but not limited to), an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the embodiments of the present disclosure, the computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0099] The computer-readable signal medium can include a propagated data signal having computer-readable program code embodied therein, either as in a baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to: electromagnetic, optical, or any suitable combination thereof. The computer-readable signal medium can be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0100] Any suitable medium can be used to transmit the program code implemented on the computer-readable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0101] Computer program code for performing operations in accordance with various aspects of the present disclosure may be written in any combination of one or more programming languages, including: object-oriented programming languages such as Java, Smalltalk, C++, PHP, Python, etc.; and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer as a stand-alone software package, partly on the user's computer, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0102] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0103] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable device, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide a process for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0104] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present disclosure having equivalent elements, modifications, omissions, combinations (e.g., schemes that cross various embodiments), adaptations, or alterations. The elements in the claims will be interpreted broadly based on the language employed in the claims and not limited to the examples described in the specification or during the implementation of the present application, and the examples will be construed as non-exclusive. Accordingly, the specification and examples are intended to be considered only as examples, with the true scope and spirit being indicated by the full scope of the following claims and their equivalents.

[0105] The foregoing description is intended to be illustrative and not restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. For instance, those of ordinary skill in the art may use other embodiments when reading the above description. Additionally, in the above detailed description, various features may be grouped together to simplify the disclosure. This should not be construed as an intention that a feature of the disclosure that is not claimed is necessary for any claim. On the contrary, the subject matter of the present invention may consist of less than all of the features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the detailed description by way of example or illustration, where each claim stands on its own as a separate embodiment, and it is contemplated that these embodiments may be combined with each other in various combinations or permutations. The scope of the present invention should be determined with reference to the appended claims and the full scope of equivalents to which those claims are entitled.

Claims

1. A medical image analysis method, characterized in that, include: receiving a medical image acquired by a medical imaging device and a medical record of a subject, wherein the medical record includes text regarding an analysis request; performing navigation processing on the medical image based on the analysis requirement to obtain a navigation trajectory, the navigation trajectory including a path or pattern of key points, or a corresponding template of structures or functional areas, wherein the structures or functional areas in the corresponding template are associated with each other according to the analysis requirement; extracting a set of image blocks along the navigation trajectory; as well as Based on the navigation trajectory and the image block set, a first learning network and a second learning network are used to determine the analysis result, wherein the first learning network is configured to: extract image features based on the image block set, and the image features are a sequence of features of each image block in the image block set; the second learning network is configured to: perform spatial or functional relationship learning based on the image block features extracted by the first learning network and the navigation trajectory, and determine the analysis result based on the spatial or functional relationship between the image block features obtained by learning.

2. The medical image analysis method according to claim 1, characterized in that The navigation trajectory is directed, the first learning network includes a convolutional neural network, and the second learning network includes one or a combination of a recurrent neural network and a recursive neural network.

3. The medical image analysis method according to claim 1, characterized in that, The analysis requirements also include the target site to be analyzed and whether the navigation is set manually or automatically.

4. The medical image analysis method according to claim 1, wherein For the problem of fatty liver classification, the navigation trajectory is a path or pattern formed by one or more key points of the liver and kidney obtained by manual annotation or automatic detection.

5. The medical image analysis method according to claim 1, characterized in that For vascular lesion detection, the navigation trajectory is the direction of the vascular segment formed by key points obtained by manual marking or automatic detection, or the complete or partial vascular centerline obtained by automatic calculation.

6. The medical image analysis method according to claim 1, characterized in that For cerebral ischemia detection, the navigation trajectory is a template corresponding to the left and right brain structures or functional areas obtained by manual delineation or automatic detection.

7. The medical image analysis method according to claim 1, wherein Also includes: extracting text features based on the text; as well as An analysis result is determined based on the text features, the navigation trajectory, and the set of image blocks using a first learning network and a second learning network.

8. The medical image analysis method according to claim 1, wherein Also includes: extracting text features based on the text; Fusing the text feature with the image feature to obtain a fused feature; The fused features are fed to the second learning network, which is configured to determine an analysis result based on the fused features.

9. The medical image analysis method according to claim 7 or 8, characterized in that The text features are extracted based on the text using a natural language processing model.

10. The medical image analysis method according to claim 8, characterized in that, The step of fusing the text feature with the image feature to obtain a fused feature further includes: The text features and the image features are merged, and the merged features are fed into a multi-layer perceptron, and the merged features are optimized by back propagation to obtain the fusion features.

11. The medical image analysis method according to any one of claims 1 to 10, characterized in that, The navigation process is a classification learning network established based on a deep learning training process.

12. A medical image analysis device, characterized in that, include: a first receiving module configured to receive a medical image acquired by a medical imaging device; a second receiving module configured to receive a medical record of a subject of the medical image, the medical record including text regarding analysis requirements; A navigation processing module configured to perform navigation processing on the medical image based on analysis requirements to obtain a navigation trajectory, where the analysis requirements include the disease to be analyzed, the navigation trajectory includes a path or pattern of key points, or a corresponding template of a structure or functional area, and each structure or functional area in the corresponding template is associated with each other for the analysis requirements; A first extraction module configured to extract a set of image patches along the navigation trajectory; And A module for determining an analysis result using a first learning network and a second learning network based on the navigation trajectory and the set of image patches, where the first learning network is configured to: extract image features based on the set of image patches, and the image features are a sequence of features of each image patch in the set of image patches; the second learning network is configured to: perform spatial or functional relationship learning based on the image patch features extracted by the first learning network and the navigation trajectory, and determine the analysis result based on the spatial or functional relationship between the learned image patch features.

13. The medical image analysis device according to claim 12, wherein The navigation trajectory is directed, the first learning network includes a convolutional neural network, and the second learning network includes one or a combination of a recurrent neural network and a recursive neural network.

14. The medical image analysis device according to claim 12, characterized in that, The analysis requirements further include the target site to be analyzed and manual or automatic navigation settings.

15. The medical image analysis device according to claim 12, characterized in that, For the problem of fatty liver grading, the navigation trajectory is a path or pattern formed by one or more key points of the liver and kidney obtained by manual annotation or automatic detection.

16. The medical image analysis device according to claim 12, wherein For vascular lesion detection, the navigation trajectory is the trend of a vascular segment formed by key points obtained by manual annotation or automatic detection, or a complete or partial vascular centerline calculated automatically.

17. The medical image analysis device according to claim 12, wherein For cerebral ischemia detection, the navigation trajectory is a corresponding template of the left and right brain corresponding structures or functional areas obtained by manual drawing or automatic detection.

18. The medical image analysis device according to claim 12, wherein Further includes: A third extraction module configured to extract text features based on the text; A module for determining an analysis result using a first learning network and a second learning network based on the text features, the navigation trajectory, and the set of image patches.

19. The medical image analysis device according to claim 12, wherein Further includes: A third extraction module configured to extract text features based on the text; A feature fusion module configured to fuse the text features with the image features to obtain fused features; And A second feeding module configured to feed the fused features into the second learning network, and the second learning network is configured to determine the analysis result based on the fused features.

20. The medical image analysis device according to claim 18 or 19, characterized in that, The text features are extracted based on the text using a natural language processing model.

21. The medical image analysis device according to claim 19, characterized in that, The feature fusion module is further configured to merge the text features with the image features, feed the merged features into a multi-layer perceptron, and optimize the merged features through backpropagation to obtain the fused features.

22. The medical image analysis device according to any one of claims 12 to 21, characterized in that, The navigation processing is a classification learning network established based on a deep learning training process.

23. A computer device, comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the operations performed by the medical image analysis method according to any one of claims 1 to 11.

24. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the operations performed by the medical image analysis method according to any one of claims 1 to 11.

Images