Medical Image Processing Method, Device, Electronic Device and Storage Medium

Abstract

The present application provides a medical image processing method, apparatus, electronic device, and storage medium. Through the processing of medical images, the reuse of medical images is promoted. While improving the utilization rate of medical images, it is also possible to improve the accuracy of disease prognosis prediction based on further analysis of medical images, thereby assisting doctors in formulating treatment plans and increasing the feasibility of clinical implementation. The medical image processing method includes: obtaining a three-dimensional medical image including a target area; inputting the three-dimensional medical image into a medical prognosis prediction model, and extracting features from each layer of the three-dimensional medical image through a first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; sequentially inputting the two-dimensional image features in the two-dimensional image feature sequence into a second network of the medical prognosis prediction model; and using the prediction data output by the second network as the prognosis prediction result of the target area.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of image processing technology, and more specifically, to a medical image processing method, device, electronic device, and storage medium. Background Art

[0002] With the popularization of multimedia technology, people use images more and more frequently in their daily lives and production activities. Medical images are a type of image. Most medical images lose their value after doctors read them. At the same time, storage space is required to store medical images, which takes up a lot of storage resources. The reuse of medical images is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a medical image processing method, device, electronic device, chip and computer-readable storage medium. By processing medical images, the reuse of medical images is promoted. While being able to improve the utilization rate of medical images, it is also possible to improve the accuracy of disease prognosis prediction based on further analysis of medical images, thereby assisting doctors in formulating treatment plans and increasing the feasibility of clinical implementation.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of the present application, a medical image processing method is provided, comprising:

[0006] Acquiring a three-dimensional medical image including a target area;

[0007] Inputting the three-dimensional medical image into the medical prognosis prediction model, extracting features from each layer of the three-dimensional medical image through the first network in the medical prognosis prediction model, and obtaining a two-dimensional image feature sequence corresponding to the three-dimensional medical image;

[0008] sequentially inputting the two-dimensional image features in the two-dimensional image feature sequence into the second network of the medical prognosis prediction model;

[0009] The predicted data output by the second network is used as the prognosis prediction result of the target area.

[0010] According to one aspect of the present application, a medical image processing device is provided, comprising:

[0011] An acquisition module, used for acquiring a three-dimensional medical image including a target area;

[0012] An input module for inputting three-dimensional medical images into a medical prognosis prediction model, and extracting features from each layer of the three-dimensional medical images through a first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical images;

[0013] The input module is further configured to sequentially input the two-dimensional image features in the two-dimensional image feature sequence into a second network of the medical prognosis prediction model;

[0014] A processing module for using the prediction data output by the second network as the prognosis prediction result of the target area.

[0015] According to one aspect of the present application, there is provided an electronic device, including: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of the above-mentioned medical image processing method.

[0016] According to one aspect of the present application, there is provided a chip, including: a processor for calling and running a computer program from a memory, so that the processor executes the steps of the above-mentioned medical image processing method.

[0017] According to one aspect of the present application, there is provided a computer-readable storage medium for storing a computer program, and the computer program enables a computer to execute the steps of the above-mentioned medical image processing method.

[0018] Based on the above technical solutions, features can be extracted from each layer of the three-dimensional medical images through the first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical images; the two-dimensional image features in the two-dimensional image feature sequence are sequentially input into the second network of the medical prognosis prediction model; and the prediction data output by the second network is used as the prognosis prediction result of the target area. Based on the first network, the three-dimensional medical images can be sliced into multiple layers of two-dimensional images and a two-dimensional image feature sequence can be formed. Based on the second network, the two-dimensional image features in the sequentially input two-dimensional image feature sequence can be fused, and prediction can be performed based on the fused features, thereby promoting the reuse of medical images. While improving the utilization rate of medical images, it is also possible to improve the accuracy of disease prognosis prediction based on further analysis of medical images. In addition, a large amount of data from different hospitals can be used to train the network model, which improves the accuracy of disease prognosis prediction based on the medical prognosis prediction model. At the same time, it can assist doctors in formulating treatment plans and increase the feasibility of clinical implementation.

[0019] Other features and advantages of the embodiments of the present application will become apparent through the following detailed description, or be learned in part through the practice of the present application.

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the present application. Brief Description of the Drawings

[0021] The drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0022] Figure 1 Schematically shows an application scenario diagram of a medical image processing method provided in an embodiment of the present application;

[0023] Figure 2 Schematically shows a flowchart of a medical image processing method according to an embodiment of the present application;

[0024] Figure 3 Schematically shows a flowchart of a training method of a medical prognosis prediction model according to an embodiment of the present application;

[0025] Figure 4 Schematically shows a schematic diagram of a medical prognosis prediction model according to an embodiment of the present application;

[0026] Figure 5 Schematically shows a block diagram of a medical image processing device according to an embodiment of the present application;

[0027] Figure 6 Shows a schematic structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. Detailed Embodiments

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present application will be more complete and will fully convey the concept of the example embodiments to those skilled in the art. The drawings are schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted.

[0029] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the example embodiments of the present application. However, those skilled in the art will realize that one or more of the specific details may be omitted in practicing the technical solutions of the present application, or other methods, components, steps, etc. may be adopted. In other cases, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the present application.

[0030] Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks, processor devices, or microcontroller devices.

[0031] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, smart healthcare, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. This application can perform multi-speaker scene recognition and multi-speaker scene recognition network training based on artificial intelligence technology.

[0032] Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a manner similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making, that is, to enable the machines to have learning capabilities.

[0033] Among them, Machine Learning (ML) is an interdisciplinary subject that involves multiple sciences such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks (such as convolutional neural networks), belief networks, reinforcement learning, transfer learning, inductive learning, and rote learning.

[0034] Artificial intelligence combined with cloud services can also realize artificial intelligence cloud services, which are generally also called AI as a Service (AIaaS). This is a current mainstream service method for artificial intelligence platforms. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI-themed mall: all developers can access and use one or more artificial intelligence services provided by the platform through the Application Programming Interface (API). Some senior developers can also use the AI frameworks and AI infrastructure provided by the platform to deploy and operate their own exclusive cloud artificial intelligence services.

[0035] Figure 1 It is an application scenario diagram of the medical image processing method provided in an embodiment, as Figure 1 shown. In this application scenario, it includes a terminal 110 and a server 120.

[0036] In some implementation manners, the medical prognosis prediction model (the first network and the second network) can be trained by the server 120. After the server 120 obtains the trained medical prognosis prediction model, it can deploy it in the image processing application. The terminal 110 can install this image processing application. When the terminal 110 obtains a three-dimensional medical image including the target area, the user can issue an image processing instruction through corresponding operations. The terminal 110 can receive the image processing instruction, use the obtained three-dimensional medical image including the target area as the image to be processed for image processing, and obtain the prognosis prediction result of the target area. For example, when the target area is a cerebral hemorrhage area, the prognosis prediction result of the target area can be the prognosis prediction result of cerebral hemorrhage (for example, 0 represents the expansion of bleeding in the cerebral hemorrhage area, 1 represents the non-expansion of bleeding in the cerebral hemorrhage area; or vice versa). When the target area is an area reflecting other types of diseases, the prognosis prediction result of the target area is similar and will not be elaborated here.

[0037] The above image processing application can be an application program for disease treatment or prevention, and the application program for disease treatment or prevention can also have functions such as data recording, audio and video playback, feature annotation, generation and modification of treatment or prevention plans, translation, data query, etc.

[0038] In some other implementation manners, the medical prognosis prediction model (the first network and the second network) can be trained by the terminal 110. After the terminal 110 obtains a three-dimensional medical image including a target area, the user can send an image processing instruction through corresponding operations, and the terminal 110 can receive the image processing instruction, perform image processing on the obtained three-dimensional medical image including the target area as the image to be processed, and obtain the prognosis prediction result of the target area. For example, when the target area is a cerebral hemorrhage area, the prognosis prediction result of the target area can be the prognosis prediction result of cerebral hemorrhage (for example, 0 represents the expansion of bleeding in the cerebral hemorrhage area, 1 represents the non-expansion of bleeding in the cerebral hemorrhage area; or vice versa).

[0039] It can be understood that the above application scenarios are only examples and do not constitute a limitation on the medical image processing method provided by the embodiments of the present application. For example, the trained medical prognosis prediction model (the first network and the second network) can be stored in the server 120, and the server 120 can receive the three-dimensional medical image including the target area sent by the terminal 110, perform image processing on the three-dimensional medical image including the target area to obtain the prognosis prediction result of the target area, and then return it to the terminal 110.

[0040] Among them, the above three-dimensional medical image including the target area can be a three-dimensional medical image such as a Computed Tomography (CT) image, a Nuclear magnetic resonance imaging (NMRI) image, etc.

[0041] The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal 110 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 110 and the server 120 can be directly or indirectly connected through wired or wireless communication methods, and the present application does not limit this.

[0042] In some embodiments, the target area includes one of the following:

[0043] Cerebral hemorrhage area, cerebral thrombosis area, hydrocephalus area, tumor area, calculus area, heart lesion area, lung cancer area.

[0044] For better understanding of the embodiments of the present application, cerebral hemorrhage is described.

[0045] Cerebral hemorrhage refers to the bleeding caused by the rupture of blood vessels in the non-traumatic cerebral parenchyma. The main causes are related to cerebrovascular diseases, that is, closely related to hyperlipidemia, diabetes, hypertension, vascular aging, smoking, etc. Patients with cerebral hemorrhage often suddenly onset due to emotional agitation or strenuous exertion. The early mortality rate is very high, and most survivors have sequelae such as varying degrees of motor impairment, cognitive impairment, speech and swallowing disorders, etc.

[0046] For cerebral hemorrhage, cranial CT scan can show the bleeding site, the size of the bleeding volume, the shape of the hematoma, whether it breaks into the ventricle, and whether there is a low-density edema zone and mass effect around the hematoma, etc. The lesions are mostly round or oval-shaped homogeneous high-density areas with clear boundaries. When there is a large amount of blood accumulation in the ventricle, it mostly shows a high-density cast and the ventricle expands. After 1 week, there is a ring enhancement around the hematoma, and after the hematoma is absorbed, it shows low density or cystic change. Dynamic CT examination can also evaluate the progress of bleeding.

[0047] For better understanding of the embodiments of the present application, the ROI area is described.

[0048] For the region of interest (ROI) area, it is to select an image area from the image, and this area is the focus of image analysis. By delineating the ROI area, the image to be processed changes from a large image to a small image area, so as to facilitate further processing and can greatly reduce the processing time.

[0049] For better understanding of the embodiments of the present application, disease prognosis prediction is described.

[0050] Disease prognosis refers to the prediction of the future development course and outcome (recovery, recurrence, deterioration, disability, complications and death, etc.) of a disease after its occurrence. Disease prognosis can utilize various information, such as clinical medical record information, and more intuitive imaging information, etc. Clinically, a large amount of information contained in medical images provides an important basis for disease prognosis.

[0051] At present, for a specific problem, three major steps of regional segmentation, feature extraction, and model establishment are required. Regional segmentation is to annotate the region of interest (ROI). Usually, we only focus on the features of certain parts in the image rather than the whole image. Therefore, it is necessary to first segment the region of interest. Then, extract the features of the region of interest, and then perform feature screening. Finally, a simple linear regression or logistic regression model is built for the screened features. The method of feature screening is to judge whether there is a significant difference in the distribution of each feature between positive and negative samples one by one.

[0052] However, at present, manual annotation of ROI is time-consuming, introduces human annotation errors, and reduces clinical acceptance. For new tasks, targeted feature screening is required before model development, with weak migration ability and long development cycles. The accuracy is limited because the linear regression model used is relatively simple and cannot handle the complex and high-dimensional relationships between various features, and may not be able to fit the training data well, resulting in model underfitting.

[0053] In order to improve the efficiency and accuracy of disease prognosis prediction, this application proposes a medical image processing solution that can develop a medical prognosis prediction model efficiently and with high quality. The first network in the medical prognosis prediction model can extract features from each layer of the three-dimensional medical image to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; the two-dimensional image features in the two-dimensional image feature sequence are sequentially input into the second network of the medical prognosis prediction model; the prediction data output by the second network is used as the prognosis prediction result of the target area. Based on the first network, the three-dimensional medical image can be sliced into multiple layers of two-dimensional images and form a two-dimensional image feature sequence. Based on the second network, the two-dimensional image features in the sequentially input two-dimensional image feature sequence can be fused, and prediction can be made based on the fused features, thus promoting the reuse of medical images. While improving the utilization rate of medical images, it can also improve the accuracy of disease prognosis prediction based on further analysis of medical images. In addition, a large amount of data from different hospitals can be used to train the network model, improving the accuracy of disease prognosis prediction based on the medical prognosis prediction model. At the same time, it can assist doctors in formulating treatment plans and increase the feasibility of clinical implementation.

[0054] It should be noted that in the embodiments of this application, cerebral hemorrhage is mainly used as an example for disease prognosis prediction, that is, the target area is the cerebral hemorrhage area, and it is predicted whether the hemorrhage in the cerebral hemorrhage area will expand. Of course, the diseases such as heart disease, cerebral thrombosis, hydrocephalus, lung cancer, tumor, and calculus can also be predicted based on the solution of the embodiments of this application, especially for vascular diseases. The embodiments of this application are not limited thereto.

[0055] The following will describe the specific implementation process of the embodiments of this application in detail.

[0056] Figure 2 FIG. 2 shows a schematic flowchart of a medical image processing method 200 according to an embodiment of the present application. The medical image processing method 200 can be executed by a device with computing and processing capabilities, such as the above-mentioned terminal 110 or server 120. Referring to Figure 2 as shown, the medical image processing method 200 may at least include S210 to S240, which are introduced in detail as follows:

[0057] In S210, a three-dimensional medical image including a target region is obtained.

[0058] In some embodiments, the target region includes, but is not limited to, one of the following:

[0059] cerebral hemorrhage region, cerebral thrombosis region, hydrocephalus region, tumor region, calculus region, heart lesion region, lung cancer region.

[0060] Specifically, the three-dimensional medical image including the target region may be one or more three-dimensional images reflecting the patient's lesion taken before and / or after surgery.

[0061] For example, when the target region is a cerebral hemorrhage region, the above three-dimensional medical image may be one or more three-dimensional images reflecting the cerebral hemorrhage lesion taken before and / or after surgery of a cerebral hemorrhage patient.

[0062] It should be noted that the present application does not limit parameters such as the size, dimensions, shape of the target region, and its position in the three-dimensional medical image. The target region can be flexibly set according to actual needs.

[0063] In some embodiments, the above three-dimensional medical image may include, but is not limited to, one of the following:

[0064] Computed Tomography (CT) image, Nuclear Magnetic Resonance Imaging (NMRI) image, Positron Emission Tomography (PET) image, Medical Ultrasonography image, Positron Emission Tomography with Computerized Tomography (PET / CT) image, Single Photon Emission Computed Tomography with Computerized Tomography (SPECT / CT) image.

[0065] CT image: Using precisely collimated X-ray beams, gamma rays, ultrasonic waves, etc., together with highly sensitive detectors, perform one cross-sectional scan after another around a certain part of the human body. It has the characteristics of fast scanning time and clear images, and can be used for the examination of various diseases.

[0066] Magnetic Resonance Imaging: Utilizes the principle of nuclear magnetic resonance. Based on the different attenuations of the released energy in different structural environments within the substance, by applying an external gradient magnetic field to detect the emitted electromagnetic waves, the position and type of the atomic nuclei constituting this object can be known, and based on this, the internal structural image of the object can be drawn. Magnetic Resonance Imaging uses a static magnetic field and a radiofrequency magnetic field to image human tissues. During the imaging process, high-contrast clear images can be obtained without using ionizing radiation or contrast agents. Magnetic Resonance Imaging can reflect disorders and early lesions of human organs from within the human molecules.

[0067] In S220, input the three-dimensional medical image into the medical prognosis prediction model, and extract features from each layer of the three-dimensional medical image through the first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image.

[0068] It should be noted that the medical prognosis prediction model can be pre-trained, that is, the model parameters of the medical prognosis prediction model are already in an optimal state.

[0069] Specifically, the first network in the medical prognosis prediction model can disassemble the three-dimensional medical image into a series of two-dimensional image slices, and extract features from each layer (two-dimensional image slice) of the three-dimensional medical image to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image.

[0070] The embodiments of the present application do not limit the specific form of the features extracted from each layer, and can be flexibly selected according to actual needs. In addition, the embodiments of the present application do not limit the specific number of layers into which the three-dimensional medical image is specifically disassembled, and can be flexibly selected according to actual needs.

[0071] For example, a brain CT has 10 layers, and each layer can extract a feature with a dimension of 2048 through the first network in the medical prognosis prediction model.

[0072] In S230, input the two-dimensional image features in the two-dimensional image feature sequence into the second network of the medical prognosis prediction model in sequence.

[0073] In S240, use the prediction data output by the second network as the prognosis prediction result of the target area.

[0074] Specifically, the second network of the medical prognosis prediction model has the ability to remember the sequence input and outputs after the sequence input is completed. The output result at this time represents the prediction of the entire sequence.

[0075] In some embodiments, the second network of the medical prognosis prediction model can fuse the two-dimensional image features in the sequentially input two-dimensional image feature sequence and make a prediction based on the fused features. That is, the above prediction data is determined based on the features after the fusion of the two-dimensional image features in the two-dimensional image feature sequence.

[0076] In some embodiments, the two-dimensional image features can be feature-fused through the second network to obtain image fusion features; based on the image fusion features, prediction data for the target region can be generated; and this prediction data can be used as the prognosis prediction result for the target region.

[0077] That is, the second network can fuse the two-dimensional image features in the sequentially input two-dimensional image feature sequence to obtain image fusion features and make a prediction based on the image fusion features.

[0078] In some embodiments, the first network is a Convolutional Neural Network (CNN), and the second network is a Recurrent Neural Network (RNN).

[0079] In some embodiments, the prediction data output by the medical prognosis prediction model can be a binary classification result. For example, when the target region is a cerebral hemorrhage region, prediction data 0 represents hemorrhage expansion, and prediction data 1 represents no hemorrhage expansion; or vice versa.

[0080] As an example, taking the brain CT of a patient as the input of a medical prognosis prediction model, the CNN (the first network) in the medical prognosis prediction model can extract the features of each layer of the brain CT scan image. The features are in the form of vectors. Taking the ResNet50 network as an example, the features of each layer of the brain CT extracted are 2048-dimensional vectors. The single-layer features of the brain CT extracted will be sequentially input into the RNN (the second network model) in the medical prognosis prediction model. The RNN can process sequential inputs and output a prediction value after the single-layer features of the brain CT are input. For example, if the brain CT has 10 layers, a 2048-dimensional feature can be extracted from each layer through the CNN. Then these 10 layers of features need to be sequentially input into the RNN, and only the value output by the RNN after the input of the 10th layer feature is taken as the final prediction value. The RNN has the ability to remember sequential inputs. Selecting the output of the RNN after the sequential input is completed can represent the prediction of the entire input sequence, that is, this prediction is for the entire brain CT. This prediction value is between 0 and 1 and represents the judgment of the medical prognosis prediction model on whether the hematoma of the cerebral hemorrhage of this patient will expand. The closer the prediction value is to 0, the less likely the brain hematoma is to expand, and the closer the prediction value is to 1, the more likely the brain hematoma is to expand. Thus, it provides a reference for doctors to formulate treatment plans.

[0081] For cerebral hemorrhage, prognosis prediction is to predict whether the volume of cerebral hemorrhage in the patient will continue to expand, because whether the volume of cerebral hemorrhage will continue to expand is related to the patient's ultimate survival or recovery status. Accurately predicting whether cerebral hemorrhage will expand can assist doctors in detecting patients at risk of hemorrhage expansion early and taking corresponding treatments early to improve the patient's survival status.

[0082] Therefore, in the embodiments of the present application, the features of each layer of the three-dimensional medical image can be extracted through the first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; the two-dimensional image features in the two-dimensional image feature sequence are sequentially input into the second network of the medical prognosis prediction model; and the prediction data output by the second network is used as the prognosis prediction result of the target area. Based on the first network, the three-dimensional medical image can be sliced into multiple layers of two-dimensional images and form a two-dimensional image feature sequence. Based on the second network, the two-dimensional image features in the sequentially input two-dimensional image feature sequence can be fused, and prediction can be made based on the fused features, thereby promoting the reuse of medical images. While improving the utilization rate of medical images, it can also improve the accuracy of disease prognosis prediction based on further analysis of medical images. In addition, a large amount of data from different hospitals can be used to train the network model, improving the accuracy of disease prognosis prediction based on the medical prognosis prediction model. At the same time, it can assist doctors in formulating treatment plans and increase the feasibility of clinical implementation.

[0083] Figure 3FIG. 0 shows a schematic flowchart of a training method 300 of a medical prognosis prediction model according to an embodiment of the present application. The training method 300 of the medical prognosis prediction model can be executed by a device with computing and processing capabilities, such as the above-mentioned terminal 110 or server 120. Refer to Figure 3 As shown, the training method 300 of the medical prognosis prediction model can at least include S310 to S320, which are introduced in detail as follows:

[0084] In S310, a training sample set is obtained, and each training sample in the training sample set includes a three-dimensional medical image including a target region and a paired medical label.

[0085] In some embodiments, the target region includes, but is not limited to, one of the following:

[0086] Intracerebral hemorrhage region, cerebral thrombosis region, hydrocephalus region, tumor region, calculus region, heart lesion region, lung cancer region.

[0087] Specifically, the three-dimensional medical image including the target region in the training sample can be one or more three-dimensional images reflecting the patient's lesion taken before and / or after surgery.

[0088] For example, when the target region is an intracerebral hemorrhage region, the three-dimensional medical image included in the above training sample can be one or more three-dimensional images reflecting the intracerebral hemorrhage lesion taken before and / or after surgery of the intracerebral hemorrhage patient.

[0089] It should be noted that the present application does not limit parameters such as the size, dimension, shape of the target region, and its position in the three-dimensional medical image, and the target region can be flexibly set according to actual needs.

[0090] In some embodiments, the three-dimensional medical image included in the above training sample can include, but is not limited to, one of the following:

[0091] Computed tomography (CT) image, nuclear magnetic resonance imaging (NMRI) image, positron emission tomography (PET) image, medical ultrasonography image, positron emission tomography / computed tomography (PET / CT) image, single photon emission computed tomography (SPECT / CT) image.

[0092] CT image: Using precisely collimated X-ray beams, γ rays, ultrasonic waves, etc., together with highly sensitive detectors, make one cross-sectional scan after another around a certain part of the human body. It has the characteristics of fast scanning time and clear images, and can be used for the examination of various diseases.

[0093] Magnetic Resonance Imaging (MRI): It utilizes the principle of nuclear magnetic resonance. Based on the different attenuations of the released energy in different structural environments within a substance, by applying an external gradient magnetic field to detect the emitted electromagnetic waves, the positions and types of the atomic nuclei that make up this object can be known, and based on this, an internal structural image of the object can be drawn. MRI uses a static magnetic field and a radiofrequency magnetic field to image human tissues. During the imaging process, high-contrast clear images can be obtained without using ionizing radiation or contrast agents. MRI can reflect the disorders and early lesions of human organs from within the human molecules.

[0094] The number of training samples in the above training sample set can be set according to requirements. For example, if the model needs to be trained 40 times during training, and 5000 training samples are used for each training, then the training sample set can include 5000 training samples, and these 5000 training samples are used for each model training. The three-dimensional medical image including the target area and the paired medical label refer to: the three-dimensional medical image including the target area is the image to be predicted, and the medical label is the predicted result expected after predicting the three-dimensional medical image including the target area using the medical prognosis prediction model.

[0095] The goal of model training is to obtain better model parameters to improve the prediction effect. During training, the three-dimensional medical image including the target area is input into the medical prognosis prediction model, and then the model parameters are adjusted according to the difference between the prediction result output by the medical prognosis prediction model and the medical label corresponding to the three-dimensional medical image including the target area, so that the prediction result obtained according to the adjusted model parameters is getting closer and closer to the medical label corresponding to the three-dimensional medical image including the target area until the model convergence condition is met, and finally a trained medical prognosis prediction model is obtained.

[0096] It should be noted that the three-dimensional medical images including the target area in the above training sample set can come from a large amount of data from different hospitals, that is, the samples are more abundant, which improves the training effect of the medical prognosis prediction model.

[0097] In some embodiments, multiple original three-dimensional medical images including the target area and parameter information of a second image processing method are obtained, where the second image processing method includes at least one of cropping, flipping, translation, scaling, and rotation; randomly using one or more of the processing methods included in the second image processing method to process some or all of the multiple original three-dimensional medical images including the target area to obtain at least one enhanced three-dimensional medical image including the target area; using the multiple original three-dimensional medical images including the target area and the at least one enhanced three-dimensional medical image including the target area as training samples to form the above training sample set. Thus, the diversity of the samples is increased, and the training effect of the network model is improved.

[0098] Specifically, the training sample set can include both multiple original three-dimensional medical images including the target region and at least one enhanced three-dimensional medical image including the target region. That is, the number of training samples included in the above training sample set can be greater than the number of multiple original three-dimensional medical images including the target region.

[0099] Since it is difficult to obtain medical image data and the amount of available data is small, if only the original data is used as training samples for training, overfitting of the medical prognosis prediction model may occur. In order to train a medical prognosis prediction model with better generalization ability, it is necessary to perform some transformations on the original data to obtain the training sample set. In this application, one or more of cropping, flipping, translation, scaling, and rotation can be randomly used to process multiple original three-dimensional medical images including the target region to obtain the above training sample set. Data augmentation is performed on the original three-dimensional medical images including the target region, increasing the diversity of samples and improving the training effect of the network model.

[0100] It should be noted that in the process of obtaining the training sample set, the cropping parameters such as the cropping size and shape of the three-dimensional medical image including the target region can be flexibly set according to requirements, the flipping parameters such as the flipping angle of the three-dimensional medical image including the target region can be flexibly set according to requirements, the translation parameters such as the translation distance of the three-dimensional medical image including the target region can be flexibly set according to requirements, the scaling parameters such as the scaling ratio of the three-dimensional medical image including the target region can be flexibly set according to requirements, and the rotation parameters such as the rotation angle of the three-dimensional medical image including the target region can be flexibly set according to requirements. This application does not limit this.

[0101] In S220, the medical prognosis prediction model is trained according to the training sample set;

[0102] Among them, during the training process, the first network extracts features from each layer of the three-dimensional medical image in the training sample to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image in the training sample, and sequentially inputs the two-dimensional image features in the two-dimensional image feature sequence into the second network; the second network fuses the two-dimensional image features in the two-dimensional image feature sequence and makes a prediction according to the fused features.

[0103] In some embodiments, the first network is a convolutional neural network (CNN), and the second network is a recurrent neural network (RNN).

[0104] For example, the first network is a 2D convolutional network such as VggNet, ResNet, DenseNet, etc.

[0105] VggNet explored the relationship between the depth of convolutional neural networks and their performance. By repeatedly stacking small 3*3 convolutional kernels and 2*2 max-pooling layers, it successfully constructed convolutional neural networks with 16 to 19 layers. VggNet divided the network into 5 segments. In each segment, multiple 3*3 convolutional networks were connected in series, followed by a max-pooling layer after each segment of convolution. Finally, there were 3 fully connected layers and a softmax layer. Softmax algorithm: "Compresses" a K-dimensional vector Z containing arbitrary real numbers into another K-dimensional real vector f(Z), such that the range of each element is between (0,1) and the sum of all elements is 1. Among them, e i represents the unit vector with 1 at the i-th position and 0 at the remaining positions.

[0106] The characteristic of Residual Network (ResNet) is that it is easy to optimize and can improve the accuracy by increasing a considerable depth. The internal residual blocks use skip connections, alleviating the problem of vanishing gradients caused by increasing depth in deep neural networks.

[0107] DenseNet can perform a concatenate (cat) operation on the input. An intuitive effect is that the feature maps learned by each layer can be directly used by all subsequent layers, which enables the reuse of features throughout the network and makes the model more concise. One advantage of DenseNet is that the network is narrower and has fewer parameters, largely due to the design of this dense block. In the dense block, the number of output feature maps of each convolutional layer is small (less than 100), rather than having a width of hundreds or thousands as in other networks. At the same time, this connection method makes the transmission of features and gradients more effective, and the network is easier to train. DenseNet is equivalent to directly connecting each layer to the input and the loss, so it can alleviate the phenomenon of vanishing gradients.

[0108] For example, the second network is a recurrent neural network such as Long Short-Term Memory (LSTM), GRU, etc.

[0109] Long Short-Term Memory (LSTM) is a type of time recurrent neural network, which is specifically designed to solve the long-term dependence problem existing in general RNNs (recurrent neural networks). All RNNs have a chain form of repeating neural network modules.

[0110] GRU is a variant of LSTM that uses an update gate and a reset gate. Basically, these two gating vectors determine which information can ultimately be the output of the gated recurrent unit. What is special about these two gating mechanisms is that they can preserve information in long sequences and will not be cleared over time or removed because they are irrelevant to the prediction.

[0111] In the embodiments of the present application, according to different requirements, the first network model (CNN model) and the second network model (RNN model) can adopt different specific implementation manners, with stronger flexibility.

[0112] Specifically, during the training process, the first network disassembles the input three-dimensional medical image including the target area into a series of two-dimensional image slices for processing. First, a 2D convolutional neural network (such as VggNet, ResNet, DenseNet, etc.) is used to extract the feature sequences of a series of two-dimensional image slices from the three-dimensional medical image including the target area. Then, a recurrent neural network (such as LSTM, GRU, etc.) is used to fuse the feature vectors of the two-dimensional image features input in sequence, and predict data is output according to the fused features. The recurrent neural network can handle sequence inputs well, integrate them (i.e., fuse them) and then output a final feature (the fused feature).

[0113] In some embodiments, the above medical prognosis prediction model can be an end-to-end model, and no additional operations are required in the process from input to output. By reducing manual preprocessing and subsequent processing, as much as possible, the model goes from the original input to the final output, giving the model more space to automatically adjust according to the data and increasing the overall fit of the model.

[0114] In some embodiments, parameter information of a first image processing method is obtained, where the first image processing method includes at least one of cropping, flipping, translation, scaling, and rotation; one or more of the processing methods included in the first image processing method are randomly used to process the three-dimensional medical images in the training samples to obtain an enhanced training sample set; and the medical prognosis prediction model is trained according to the enhanced training sample set.

[0115] In some embodiments, before training the medical prognosis prediction model according to the training sample set, one or more of cropping, flipping, translation, scaling, and rotation are randomly used to process the three-dimensional medical images in the training sample set. Thus, the diversity of the three-dimensional medical images in the training sample set is increased, and the training effect of the network model is improved.

[0116] Since it is difficult to obtain medical image data and the amount of available data is small, if only the three-dimensional medical images in the training sample set are used as the input for training the medical prognosis prediction model, overfitting of the medical prognosis prediction model may occur. In order to train a medical prognosis prediction model with better generalization ability, it is necessary to perform some transformations on the three-dimensional medical images in the training sample set to increase the diversity of the three-dimensional medical images in the training sample set. In this application, before inputting the three-dimensional medical image into the first network, one or more of cropping, flipping, translation, scaling, and rotation can be randomly used to process the three-dimensional medical images in the training sample set, increasing the diversity of the three-dimensional medical images in the training sample set and improving the training effect of the network model.

[0117] It should be noted that the cropping parameters such as the cropping size and shape of the three-dimensional medical images in the training sample set can be flexibly set according to requirements, the flipping parameters such as the flipping angle of the three-dimensional medical images in the training sample set can be flexibly set according to requirements, the translation parameters such as the translation distance of the three-dimensional medical images in the training sample set can be flexibly set according to requirements, the scaling parameters such as the scaling ratio of the three-dimensional medical images in the training sample set can be flexibly set according to requirements, and the rotation parameters such as the rotation angle of the three-dimensional medical images in the training sample set can be flexibly set according to requirements. This application does not limit this.

[0118] In some embodiments, before model training, the model parameters of the first network and / or the second network are initialized using a random initialization method.

[0119] In some embodiments, the first preset value and / or the second preset value are read from the database; the first preset value is used as the initial parameter of the first network, and / or, the second preset value is used as the initial parameter of the second network.

[0120] In some embodiments, the above first preset value is obtained from the training of the first network in other image processing tasks, and / or, the above second preset value is obtained from the training of the second network in other image processing tasks. That is, the network parameters trained by the first network in other medical image processing tasks can be used as the initial parameters of the first network, thereby accelerating the model convergence rate. Similarly, the convergence parameters of the second network can be obtained from the training of the second network in other image processing tasks. That is, the network parameters trained by the second network in other medical image processing tasks can be used as the initial parameters of the second network, thereby accelerating the model convergence rate.

[0121] In some embodiments, the cross-entropy loss function and the adaptive gradient descent algorithm are used to optimize the model parameters of the medical prognosis prediction model. Of course, other methods can also be used to optimize the model parameters of the medical prognosis prediction model. This application does not limit this.

[0122] In some embodiments, a cross-entropy loss amount is determined based on prediction data output by a medical prognosis prediction model and medical labels; and based on the cross-entropy loss amount, an adaptive gradient descent algorithm is used to optimize the model parameters of a first network and a second network in the medical prognosis prediction model.

[0123] Cross-entropy can measure the degree of difference between two different probability distributions in the same random variable, which in machine learning represents the difference between the true probability distribution and the predicted probability distribution. The smaller the value of the cross-entropy, the better the model prediction effect.

[0124] The adaptive gradient descent algorithm can be, for example, the Adam algorithm, which uses momentum and an adaptive learning rate to accelerate the convergence speed.

[0125] In some embodiments, a medical prognosis prediction model trained based on training samples can be verified to obtain the optimal parameters of the medical prognosis prediction model. The verification process can specifically include:

[0126] Obtain a verification sample set, where each verification sample in the verification sample set includes a three-dimensional medical image including a target region and a paired medical label;

[0127] In the case where the medical prognosis prediction model fits the training samples, the three-dimensional medical image in the verification sample is input into the medical prognosis prediction model, and each layer of the three-dimensional medical image in the verification sample is subjected to feature extraction through the first network to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image in the verification sample, and the two-dimensional image features in the two-dimensional image feature sequence are sequentially input into the second network;

[0128] Based on the prediction data output by the second network and the medical labels in the verification samples, determine the cross-entropy loss amount corresponding to the verification samples; and based on the cross-entropy loss amount corresponding to the verification samples, select the optimal parameters of the medical prognosis prediction model from multiple sets of convergence parameters obtained during the training process of the medical prognosis prediction model.

[0129] The verification sample set is used to select the network model parameters with the best generalization performance during the training process to improve the prediction effect. In the case where the medical prognosis prediction model fits the training samples after multiple iterations, the three-dimensional medical image including the target region in the verification sample set can be input into the medical prognosis prediction model to select the network model parameters with the best generalization performance. This avoids the situations of underfitting or overfitting in model training and ensures the accuracy of model training.

[0130] The number of validation samples in the above-mentioned validation sample set can be set according to requirements. The three-dimensional medical images including the target area and the paired medical labels in the validation sample set refer to: the three-dimensional medical image of the target area is the image to be predicted, and the medical label is the predicted result expected after predicting the three-dimensional medical image including the target area using the medical prognosis prediction model.

[0131] It should be noted that the validation sample set is similar to the training sample set. The three-dimensional medical images including the target area in the validation sample set can come from a large amount of data from different hospitals. That is, the samples are more abundant, improving the training effect of the medical prognosis prediction model.

[0132] In some embodiments, one or more of cropping, flipping, translation, scaling, and rotation can be randomly used to process multiple original three-dimensional medical images including the target area to obtain the above-mentioned validation sample set. Thus, the diversity of the samples is increased, and the training effect of the network model is improved.

[0133] Since it is difficult to obtain medical image data and the number of available data is small, if only the original data is used as the validation sample for verification, it may not be possible to accurately select the network model parameters with the best generalization performance. In order to select a medical prognosis prediction model with better generalization ability, some transformations need to be performed on the original data to obtain the validation sample set. In this application, one or more of cropping, flipping, translation, scaling, and rotation can be randomly used to process multiple original three-dimensional medical images including the target area to obtain the above-mentioned validation sample set. The original three-dimensional medical images including the target area are data-augmented, increasing the diversity of the samples and improving the training effect of the network model.

[0134] It should be noted that during the process of obtaining the validation sample set, the cropping parameters such as the cropping size and shape of the three-dimensional medical image can be flexibly set according to requirements, the flipping parameters such as the flipping angle of the three-dimensional medical image can be flexibly set according to requirements, the translation parameters such as the translation distance of the three-dimensional medical image can be flexibly set according to requirements, the scaling parameters such as the scaling ratio of the three-dimensional medical image can be flexibly set according to requirements, and the rotation parameters such as the rotation angle of the three-dimensional medical image can be flexibly set according to requirements. This application does not limit this.

[0135] In some embodiments, the number of validation samples in the validation sample set and the number of training samples in the training sample set can be set in a certain proportion. For example, the number of validation samples in the validation sample set and the number of training samples in the training sample set can be set in a ratio of 2:8.

[0136] For intracerebral hemorrhage, prognosis prediction is to predict whether the intracerebral hemorrhage volume of the patient will continue to expand, because whether the intracerebral hemorrhage volume will continue to expand is related to the patient's ultimate survival or recovery status. Accurately predicting whether intracerebral hemorrhage will expand can assist doctors in detecting patients at risk of hemorrhage expansion early and taking corresponding treatments early to improve the patient's survival status.

[0137] For example, taking intracerebral hemorrhage as an example, the prediction data output by the medical prognosis prediction model can be a binary classification result. For example, 0 represents hemorrhage expansion, and 1 represents no hemorrhage expansion; or vice versa.

[0138] In the embodiments of the present application, the three-dimensional medical images of the target area in the training samples and the three-dimensional medical images including the target area in the above medical image processing method 200 are medical images for the same disease (i.e., the same type of target area). For example, a medical prognosis prediction model trained based on three-dimensional medical images including the intracerebral hemorrhage area is used to process three-dimensional medical images including the intracerebral hemorrhage area to obtain a prognosis prediction result for the intracerebral hemorrhage area.

[0139] In the embodiments of the present application, the medical prognosis prediction model has strong fitting ability and high prediction accuracy. It can utilize a large amount of data from different hospitals without underfitting, improving the prediction accuracy of the medical prognosis prediction model.

[0140] The above medical prognosis prediction model can be, for example, as Figure 4 shown, the medical prognosis prediction model includes a first network and a second network. The first network of the medical prognosis prediction model can include multiple CNN networks, and each CNN network corresponds to a two-dimensional image. That is, the three-dimensional medical image including the target area can be sliced into multiple layers of two-dimensional images through multiple CNN networks, and feature extraction is performed on each layer of the three-dimensional medical image to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; the two-dimensional image features in the two-dimensional image feature sequence are sequentially input into the second network of the medical prognosis prediction model. The second network model can be an RNN network. The second network fuses the two-dimensional image features in the sequentially input two-dimensional image feature sequence and outputs prediction data according to the fused features. As Figure 4 shown, the cross-entropy loss amount can be output based on the prediction data and the medical label, and according to the cross-entropy loss amount, the model parameters of the first network and the second network are optimized using the adaptive gradient descent algorithm.

[0141] Therefore, in the embodiments of the present application, the first network in the medical prognosis prediction model can be used to extract features from each layer of the three-dimensional medical image, so as to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; the two-dimensional image features in the two-dimensional image feature sequence are sequentially input into the second network of the medical prognosis prediction model; and the prediction data output by the second network is used as the prognosis prediction result of the target area. Based on the first network, the three-dimensional medical image can be sliced into multiple layers of two-dimensional images and a two-dimensional image feature sequence can be formed. Based on the second network, the two-dimensional image features in the sequentially input two-dimensional image feature sequence can be fused, and prediction can be performed based on the fused features, thereby promoting the reuse of medical images. While improving the utilization rate of medical images, it can also improve the accuracy of disease prognosis prediction based on further analysis of medical images. In addition, a large amount of data from different hospitals can be used to train the network model, which improves the accuracy of disease prognosis prediction based on the medical prognosis prediction model. At the same time, it can assist doctors in formulating treatment plans and increase the feasibility of clinical implementation.

[0142] In the embodiments of the present application, the overall process of disease prediction is more concise. In the testing stage, no manually labeled ROI is required, which is more convenient to use and increases the feasibility of clinical implementation. In addition, no related operations such as feature extraction and screening are required, and an end-to-end training method is used, which is highly automated.

[0143] In the embodiments of the present application, according to different requirements, the first network (CNN model) and the second network (RNN model) can adopt different specific implementation manners, with stronger flexibility.

[0144] As described above in conjunction with Figures 2 to 4 , the method embodiments of the present application have been described in detail. Below, in conjunction with Figure 5 , the device embodiments of the present application will be described in detail. It should be understood that the device embodiments and the method embodiments correspond to each other, and similar descriptions can refer to the method embodiments.

[0145] Figure 5 The block diagram of a medical image processing device according to an embodiment of the present application is schematically shown. The medical image processing device can adopt software units or hardware units, or a combination of both to become a part of a computer device. As Figure 5 shown, the medical image processing device 400 provided in the embodiments of the present application may specifically include:

[0146] An acquisition module 410, configured to acquire a three-dimensional medical image including a target area;

[0147] An input module 420 for inputting a three-dimensional medical image into a medical prognosis prediction model, extracting features from each layer of the three-dimensional medical image through a first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image;

[0148] The input module 420 is further configured to sequentially input the two-dimensional image features in the two-dimensional image feature sequence into a second network of the medical prognosis prediction model;

[0149] A processing module 430 for using the prediction data output by the second network as the prognosis prediction result of the target area.

[0150] In one embodiment, the processing module 430 is specifically configured to:

[0151] Perform feature fusion on the two-dimensional image features through the second network to obtain image fusion features;

[0152] Generate prediction data for the target area based on the image fusion features;

[0153] Use the prediction data as the prognosis prediction result of the target area.

[0154] In one embodiment, the acquisition module 410 is further configured to acquire a training sample set, and each training sample in the training sample set includes a three-dimensional medical image including a target area and a paired medical label;

[0155] The processing module 430 is further configured to train the medical prognosis prediction model according to the training sample set;

[0156] Wherein, during the training process, the first network extracts features from each layer of the three-dimensional medical image in the training sample to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image in the training sample, and sequentially inputs the two-dimensional image features in the two-dimensional image feature sequence into the second network; the second network fuses the two-dimensional image features in the two-dimensional image feature sequence and makes a prediction according to the fused features.

[0157] In one embodiment, the acquisition module 410 is further configured to acquire parameter information of a first image processing method, where the first image processing method includes at least one of cropping, flipping, translation, scaling, and rotation;

[0158] The processing module 430 is further configured to randomly use one or more of the processing methods included in the first image processing method to process the three-dimensional medical image in the training sample before training the medical prognosis prediction model according to the training sample set to obtain an enhanced training sample set; and train the medical prognosis prediction model according to the enhanced training sample set.

[0159] In one embodiment, the obtaining module 410 is further configured to obtain a plurality of original three-dimensional medical images including a target region;

[0160] The obtaining module 410 is further configured to obtain parameter information of a second image processing method, where the second image processing method includes at least one of cropping, flipping, translation, scaling, and rotation;

[0161] The processing module 430 is further configured to randomly adopt one or more of the processing methods included in the second image processing method to process some or all of the plurality of original three-dimensional medical images including the target region, to obtain at least one enhanced three-dimensional medical image including the target region; and use the plurality of original three-dimensional medical images including the target region and the at least one enhanced three-dimensional medical image including the target region as training samples to form the training sample set.

[0162] In one embodiment, the processing module 430 is further configured to determine a cross-entropy loss amount according to the prediction data output by the medical prognosis prediction model and the medical label;

[0163] The processing module 430 is further configured to optimize the model parameters of the first network and the second network in the medical prognosis prediction model by using the adaptive gradient descent algorithm according to the cross-entropy loss amount.

[0164] In one embodiment, the obtaining module 410 is further configured to read a first preset value and / or a second preset value from a database;

[0165] The processing model 430 is further configured to use the first preset value as the initial parameter of the first network and / or use the second preset value as the initial parameter of the second network.

[0166] In one embodiment, the obtaining model 410 is further configured to obtain a validation sample set, where each validation sample in the validation sample set includes a three-dimensional medical image including a target region and a paired medical label;

[0167] The input module 420 is further configured to, when the medical prognosis prediction model fits the training samples, input the three-dimensional medical image in the validation sample into the medical prognosis prediction model, extract features from each layer of the three-dimensional medical image in the validation sample through the first network, to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image in the validation sample, and sequentially input the two-dimensional image features in the two-dimensional image feature sequence into the second network;

[0168] The processing module 430 is further configured to determine a cross-entropy loss amount corresponding to the validation sample according to the prediction data output by the second network and the medical label in the validation sample;

[0169] The processing module 430 is further configured to select the optimal parameters of the medical prognosis prediction model from multiple sets of convergence parameters obtained during the training process of the medical prognosis prediction model according to the cross-entropy loss amount corresponding to the verification sample.

[0170] In one embodiment, the target region includes one of the following:

[0171] Cerebral hemorrhage region, cerebral thrombosis region, hydrocephalus region, tumor region, calculus region, heart lesion region, lung cancer region.

[0172] In one embodiment, the first network in the medical prognosis prediction model is a convolutional neural network, and the second network in the medical prognosis prediction model is a recurrent neural network.

[0173] The specific implementation of each module in the medical image processing device provided in the embodiments of the present application can refer to the content in the above-mentioned medical image processing method, which will not be elaborated here.

[0174] Each module in the above-mentioned medical image processing device can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned each module can be embedded in the processor in the computer device in the form of hardware or independent of the processor, or stored in the memory in the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned each module.

[0175] Figure 6 The structural diagram of the computer system of the electronic device for implementing the embodiments of the present application is shown. It should be noted that Figure 6 The computer system 500 of the electronic device shown is only an example, and should not bring any limitation to the functions and usage scope of the embodiments of the present application.

[0176] As Figure 6 shown, the computer system 500 includes a central processing unit (CPU) 501, which can execute various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 into the random access memory (RAM) 503. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other through a bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0177] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 510 as needed so that a computer program read therefrom can be installed into the storage section 508 as needed.

[0178] Specifically, according to an embodiment of the present application, the process described in the above flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the method shown in the above flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 509, and / or installed from the removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the apparatus of the present application are executed.

[0179] In one embodiment, an electronic device is further provided, including:

[0180] a processor; and

[0181] a memory for storing executable instructions of the processor;

[0182] wherein the processor is configured to execute the steps in the above method embodiments by executing the executable instructions.

[0183] In one embodiment, a computer device is further provided, including a memory and a processor, and a computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0184] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0185] It should be noted that the computer-readable storage medium described in this application can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic disk storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. In the application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.

[0186] This embodiment is only used to illustrate this application. The selection of the software and hardware platform architecture, development environment, development language, message acquisition source, etc. of this embodiment can be changed. Based on the technical solution of this application, any improvement and equivalent transformation made to a certain part according to the principle of this application should not be excluded from the protection scope of this application.

[0187] It should be noted that the terms used in the embodiments of this application and the appended claims are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of this application.

[0188] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of this application.

[0189] If it is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks, or optical discs that can store program codes.

[0190] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the devices, apparatuses, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0191] In several embodiments provided in this application, it should be understood that the disclosed electronic devices, apparatuses, and methods can be implemented in other ways.

[0192] For example, the division of units, modules, or components in the device embodiments described above is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units, modules, or components can be combined or integrated into another system, or some units, modules, or components can be ignored or not executed.

[0193] Again, for example, the units / modules / components described as separate / display components may or may not be physically separated, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units / modules / components can be selected according to actual needs to achieve the objectives of the embodiments of this application.

[0194] Finally, it should be noted that the couplings, direct couplings, or communication connections shown or discussed above with each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can be in electrical, mechanical, or other forms.

[0195] The above content is only the specific implementation manner of the embodiments of the present application. However, the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the embodiments of the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A medical image processing method, characterized in that, Including: Obtain a three-dimensional medical image including a target region; Input the three-dimensional medical image into a medical prognosis prediction model, and extract features from each layer of the three-dimensional medical image through a first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; Input the two-dimensional image features in the two-dimensional image feature sequence into a second network of the medical prognosis prediction model in sequence; Use the prediction data output by the second network as the prognosis prediction result of the target region; The step of using the prediction data output by the second network as the prognosis prediction result of the target region includes: Perform feature fusion on the two-dimensional image features through the second network to obtain an image fusion feature; Generate prediction data of the target region based on the image fusion feature; Use the prediction data as the prognosis prediction result of the target region.

2. The method according to claim 1, wherein The method further includes: Obtain a training sample set, where each training sample in the training sample set includes a three-dimensional medical image including a target region and a paired medical label; Train the medical prognosis prediction model according to the training sample set; Wherein, during the training process, the first network extracts features from each layer of the three-dimensional medical image in the training sample to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image in the training sample, and inputs the two-dimensional image features in the two-dimensional image feature sequence into the second network in sequence; the second network fuses the two-dimensional image features in the two-dimensional image feature sequence and makes a prediction according to the fused features.

3. The method according to claim 2, characterized in that, The step of training the medical prognosis prediction model according to the training sample set includes: Obtain parameter information of a first image processing method, where the first image processing method includes at least one of cropping, flipping, translation, scaling, and rotation; Randomly use one or more of the processing methods included in the first image processing method to process the three-dimensional medical image in the training sample to obtain an enhanced training sample set; Train the medical prognosis prediction model according to the enhanced training sample set.

4. The method according to claim 2, wherein The step of obtaining the training sample set includes: Obtain multiple original three-dimensional medical images including a target region; Obtain parameter information of a second image processing method, where the second image processing method includes at least one of cropping, flipping, translation, scaling, and rotation; Randomly use one or more of the processing methods included in the second image processing method to process some or all of the multiple original three-dimensional medical images including a target region to obtain at least one enhanced three-dimensional medical image including a target region; Use the multiple original three-dimensional medical images including a target region and the at least one enhanced three-dimensional medical image including a target region as training samples to form the training sample set.

5. The method according to claim 2, wherein The step of training the medical prognosis prediction model according to the training sample set includes: Determine a cross-entropy loss amount according to the prediction data output by the medical prognosis prediction model and the medical label; According to the cross-entropy loss amount, the model parameters of the first network and the second network in the medical prognosis prediction model are optimized by using an adaptive gradient descent algorithm.

6. The method according to claim 2, wherein The training of the medical prognosis prediction model according to the training sample set includes: Reading a first preset value and / or a second preset value from a database; Using the first preset value as the initial parameter of the first network, and / or using the second preset value as the initial parameter of the second network.

7. The method according to claim 2, characterized in that, The method further includes: Obtaining a validation sample set, where each validation sample in the validation sample set includes a three-dimensional medical image including a target area and a paired medical label; When the medical prognosis prediction model fits the training samples, inputting the three-dimensional medical image in the validation sample into the medical prognosis prediction model, extracting features from each layer of the three-dimensional medical image in the validation sample through the first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image in the validation sample, and sequentially inputting the two-dimensional image features in the two-dimensional image feature sequence into the second network; Determining the cross-entropy loss amount corresponding to the validation sample according to the prediction data output by the second network and the medical label in the validation sample; Selecting the optimal parameters of the medical prognosis prediction model from multiple sets of convergence parameters obtained during the training of the medical prognosis prediction model according to the cross-entropy loss amount corresponding to the validation sample.

8. The method according to claim 1, wherein The first network in the medical prognosis prediction model is a convolutional neural network, and the second network in the medical prognosis prediction model is a recurrent neural network.

9. A medical image processing device, characterized in that, It includes: An acquisition module for acquiring a three-dimensional medical image including a target area; An input module for inputting the three-dimensional medical image into a medical prognosis prediction model, and extracting features from each layer of the three-dimensional medical image through the first network in the medical prognosis prediction model to obtain a two-dimensional image feature sequence corresponding to the three-dimensional medical image; The input module is further configured to sequentially input the two-dimensional image features in the two-dimensional image feature sequence into the second network of the medical prognosis prediction model; A processing module for using the prediction data output by the second network as the prognosis prediction result of the target area; The using the prediction data output by the second network as the prognosis prediction result of the target area includes: Performing feature fusion on the two-dimensional image features through the second network to obtain image fusion features; Generating prediction data of the target area based on the image fusion features; Using the prediction data as the prognosis prediction result of the target area.

10. An electronic device, characterized in that, It includes: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images