Image processing method, device and equipment for medical image and storage medium

Abstract

The application relates to an image processing method, device and equipment for medical images and a storage medium, and relates to the technical field of medical treatment. The method comprises the following steps: calling a first channel network to process a first sample image, obtaining a prediction type result corresponding to the first sample image and an image feature; calling a second channel network to process a second sample image and the image feature of the first sample image, and obtaining a prediction classification result; based on the prediction type result, the prediction classification result, a type label of the first sample image and a type label of the second sample, training the first channel network and the second channel network, and obtaining a channel network after the training of the first channel network is completed, and using the channel network to construct an image classification model. Through the above method, the overfitting phenomenon caused by the limited number of samples in a small sample learning scene is avoided, the training effect of the model is improved, and the classification accuracy of the obtained image classification model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the medical technology field, and in particular to an image processing method and device for medical images, an equipment and a storage medium. BACKGROUND

[0002] With the development of science and technology, in order to facilitate the extraction of information from medical images, machine learning algorithms are introduced into medical image processing.

[0003] In the related art, a convolutional neural network can be trained by a large number of medical sample images and corresponding type labels, so that the neural network model obtained by training can realize the classification of input medical images.

[0004] However, the classification accuracy of the neural network model in the above technology depends on the number and quality of the medical sample images. In the case of small sample learning, that is, in the case of small sample learning, overfitting phenomenon is easily created, so that the application range of the neural network model obtained by training is limited, and the accuracy is low. SUMMARY

[0005] The embodiments of the present application provide an image processing method, device, equipment and storage medium for medical images, which can improve the training effect of the model and improve the classification accuracy of the obtained image classification model. The technical scheme is as follows:

[0006] On the one hand, an image processing method for medical images is provided, the method comprising:

[0007] calling a first channel network to process a first sample image, obtaining a predicted type result corresponding to the first sample image and an image feature of the first sample image; the predicted type result is used to indicate the type to which the first sample image belongs;

[0008] calling a second channel network to process a second sample image and the image feature of the first sample image, obtaining a predicted classification result of the second sample image compared with the first sample image; the predicted classification result is used to indicate whether the second sample image and the first sample image are images of the same type;

[0009] training the first channel network and the second channel network based on the predicted type result, the predicted classification result, a type label of the first sample image and a type label of the second sample;

[0010] The first channel network is used to construct an image classification model, and the image classification model is used to predict the type to which a target medical image belongs.

[0011] In another aspect, an image processing apparatus for medical images is provided, the apparatus comprising:

[0012] a first processing module configured to invoke a first channel network to process a first sample image to obtain a predicted type result corresponding to the first sample image and an image feature of the first sample image, the predicted type result being indicative of a type to which the first sample image belongs;

[0013] a second processing module configured to invoke a second channel network to process a second sample image and the image feature of the first sample image to obtain a predicted classification result of the second sample image relative to the first sample image, the predicted classification result being indicative of whether the second sample image and the first sample image belong to a same type of images;

[0014] a network training module configured to train the first channel network and the second channel network based on the predicted type result, the predicted classification result, a type label of the first sample image, and a type label of the second sample image;

[0015] wherein a channel network obtained after the first channel network is trained is used to construct an image classification model, the image classification model being configured to predict a type to which a target medical image belongs.

[0016] In a possible implementation, the second channel network comprises n feature extraction layers connected layer by layer, and the first channel network comprises m feature extraction layers connected layer by layer, where 2≤n≤m, and n and m are positive integers.

[0017] In response to the first target feature extraction layer being a first feature extraction layer of the second channel network, an input of the first target feature extraction layer is the second sample image.

[0018] In response to the first target feature extraction layer being an i-th feature extraction layer of the second channel network, an input of the first target feature extraction layer comprises an output result of an (i-1)-th feature extraction layer of the second channel network and an output result of an (i-1)-th feature extraction layer of the first channel network, where 2≤i≤n.

[0019] In a possible implementation, in response to the second target feature extraction layer being a first feature extraction layer of the first channel network, an input of the second target feature extraction layer is the first sample image.

[0020] In response to the second target feature extraction layer being an lth feature extraction layer of the first channel network, an input of the second target feature extraction layer comprises an output result of an (l-1)th feature extraction layer, 2≤l≤m.

[0021] In a possible implementation, the network training module comprises:

[0022] The first parameter updating submodule is configured to perform parameter updating on the second channel network based on a difference between the predicted classification result and a classification result label, the classification result label being a classification result determined based on a type label of the first sample image and a type label of the second sample image.

[0023] The second parameter updating submodule is configured to perform parameter updating on the first channel network based on a difference between the predicted type result and the type label of the first sample image, and a difference between the predicted classification result and the classification result label.

[0024] In a possible implementation, the first parameter updating submodule is configured to calculate a function value of a first loss function based on the difference between the predicted classification result and the classification result label.

[0025] The second channel network is updated in parameter based on the function value of the first loss function.

[0026] In a possible implementation, the second parameter updating submodule is configured to calculate a function value of a first loss function based on the difference between the predicted classification result and the classification result label.

[0027] A function value of a second loss function is calculated based on a difference between the predicted classification result and the type label of the first sample image.

[0028] The first channel network is updated in parameter based on the function value of the first loss function and the function value of the second loss function.

[0029] In a possible implementation, the second parameter updating submodule is configured to perform weighted summation on the function value of the first loss function and the function value of the second loss function to obtain a weighted summation result.

[0030] The first channel network is updated in parameter based on the weighted result.

[0031] In a possible implementation, the apparatus further comprises:

[0032] The first data enhancement module is configured to perform data enhancement on the first sample image to obtain the first sample image after data enhancement.

[0033] The first processing module is configured to call a first channel network to process the first sample image after data augmentation, and obtain the prediction type result corresponding to the first sample image.

[0034] In a possible implementation, the apparatus further includes:

[0035] The second data augmentation module is configured to perform data augmentation on the second sample image to obtain the second sample image after data augmentation.

[0036] The second processing module is configured to call a second channel network to process the second sample image after data augmentation and the image features of the first sample image, and obtain the prediction classification result of the second sample image compared with the first sample image.

[0037] In a possible implementation, the data augmentation includes at least one of random rotation and random translation.

[0038] In a possible implementation, the apparatus further includes:

[0039] The first data augmentation module is configured to perform data augmentation on the target medical image to obtain x data augmentation results corresponding to the target medical image.

[0040] The first acquisition module is configured to input the x random augmentation results into the image classification model in sequence to obtain the prediction type results corresponding to the x random augmentation results, respectively.

[0041] The second acquisition module is configured to acquire the prediction type result of the target medical image based on the prediction type results corresponding to the x random augmentation results, respectively.

[0042] In another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the above-mentioned image processing method for medical images.

[0043] In another aspect, a computer readable storage medium is provided, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned image processing method for medical images.

[0044] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image processing method for medical images provided in various optional implementation manners described above.

[0045] The technical solutions provided in the present application can include the following beneficial effects:

[0046] The image processing method for medical images provided in the embodiments of the present application trains the first channel network and the second channel network by using the first sample image and the second sample image, so that the prediction classification result obtained based on the second channel network can assist the training of the first channel network, and the classification accuracy of the image classification model obtained based on the trained first channel network is improved.

[0047] Meanwhile, based on the network training of two sample images at the same time, the number of learnable sample images can be exponentially increased, so that the overfitting phenomenon caused by the limited number of sample images in the small sample learning scene is avoided, and the training effect of the model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] Figure 1 FIG. 1 shows a schematic diagram of a system architecture of the image processing method for medical images provided in an exemplary embodiment of the present application;

[0050] Figure 2 FIG. 1 shows a flowchart of the image processing method for medical images provided in an exemplary embodiment of the present application;

[0051] Figure 3 FIG. 1 is a framework diagram of an image classification model generation and image classification according to an exemplary embodiment;

[0052] Figure 4 FIG. 1 shows a flowchart of the image processing method for medical images provided in an exemplary embodiment of the present application;

[0053] Figure 5 FIG. 1 shows a schematic diagram of the first channel network and the second channel network according to an exemplary embodiment of the present application;

[0054] Figure 6 FIG. 1 shows a schematic diagram of the backbone network according to an exemplary embodiment of the present application;

[0055] Figure 7 A block diagram of an image processing apparatus for medical images is shown according to an example embodiment of the present application;

[0056] Figure 8 A structural block diagram of a computer device is shown according to an example embodiment of the present application;

[0057] Figure 9 A structural block diagram of a computer device is shown according to an example embodiment of the present application. DETAILED DESCRIPTION

[0058] The example embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, and the description which will be made is specific to embodiments of the present application. The following description is not meant to limit the application to all of the embodiments set forth herein. Rather, the following description is meant to provide examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] It should be understood that the “multiple” mentioned herein refers to two or more. The “and / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character “ / ” generally represents that the associated objects before and after are in an “or” relationship.

[0060] The embodiments of the present application provide an image processing method for medical images, which can improve the image classification accuracy. The present application relates to artificial intelligence technology and machine learning technology;

[0061] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0062] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields. The display device shown in the present application mainly relates to computer vision technology and machine learning / deep learning and other fields.

[0063] Machine learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0064] Figure 1 A schematic diagram of a system architecture for an image processing method for medical images provided by an example embodiment of the present application is shown, as shown in the figure, the system includes a computer device 110 and a medical image acquisition device 120. Figure 1

[0065] The computer device 110 can be implemented as a terminal or a server. When the computer device 110 is implemented as a server, the computer device 110 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Basic cloud computing services such as platform. When the computer device 110 is implemented as a terminal, the computer device 110 can be a smartphone, tablet computer, laptop computer, desktop computer, etc.

[0066] ​The medical image acquisition device 120 is a device with a medical image acquisition function. For example, the medical image acquisition device can be a CT (Computed Tomography) detector, a nuclear magnetic resonance detector, a positron emission computed tomography detector, a cardiac magnetic resonance detector, or any other device with an image acquisition device.

[0067] Optionally, the system includes one or more computer devices 110 and one or more medical image acquisition devices 120. The number of computer devices 110 and medical image acquisition devices 120 is not limited in the embodiments of the present application.

[0068] The medical image acquisition device 120 and the computer device 110 are connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0069] Optionally, the wireless network or the wired network uses standard communication technologies and / or protocols. The network is usually the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network of any combination. In some embodiments, technologies and / or formats such as Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In other embodiments, custom and / or proprietary data communication technologies can be used instead of or in addition to the above data communication technologies. The present application does not limit this.

[0070] Figure 2 A flowchart of an image processing method for medical images provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, which can be implemented as a server or a terminal as shown in Figure 1 Figure 2 ​As shown, the image processing method for medical images can include the following steps:

[0071] In step 210, a first channel network is invoked to process the first sample image, obtain a predicted type result corresponding to the first sample image and an image feature of the first sample image; the predicted type result is used to indicate the type to which the first sample image belongs.

[0072] In step 220, a second channel network is invoked to process the second sample image and the image feature of the first sample image, and obtain a predicted classification result of the second sample image relative to the first sample image; the predicted classification result is used to indicate whether the second sample image and the first sample image are images of the same type.

[0073] In the embodiments of the present application, the sample image set includes a plurality of sample images, and each two sample images can form a sample image pair. The first sample image and the second sample image are two sample images in the same sample image pair. Each sample image has a corresponding type label, and the type label of each sample image is used to indicate the type to which the sample image belongs. The sample images in the sample image set can be medical sample images, and each medical sample image is a medical sample image corresponding to the same disease. The type to which each sample image belongs can be used to assist medical personnel in determining the image type, assisting in disease diagnosis, such as recurrence, risk of illness, and the like. For example, the type to which the sample image belongs can be used to indicate whether the disease has recurred or not, or the type to which the sample image belongs can also be used to indicate the risk of recurrence of the disease, such as no risk of recurrence, low risk of recurrence, medium risk of recurrence, or high risk of recurrence.

[0074] In step 230, the first channel network and the second channel network are trained based on the predicted type result, the predicted classification result, the type label of the first sample image, and the type label of the second sample.

[0075] The image processing method for medical images provided in the embodiments of the present application can be applied to a small sample learning scenario. Small sample learning (Few-shot learning) is a branch of the field of machine learning, aiming to solve machine learning tasks with limited data. In small sample learning, fewer samples can be used, and using traditional machine learning methods can easily cause overfitting, reducing the generalization performance of the model algorithm. Small sample learning expects to use as few samples as possible while ensuring good performance.

[0076] In the embodiments of the present application, two different sample images in the sample image set need to be input in a round of training process, therefore, the images in the sample image set can be randomly grouped first to obtain at least two first sample image pairs, and after the iterative training of the first channel network and the second channel network based on the at least two first sample image pairs is performed according to the training process of steps 210 to 230, if the training completion condition is not reached, the images in the sample image set can be randomly grouped again to obtain at least two second sample image pairs, and the first channel network and the second channel network are trained based on the at least two second sample image pairs according to the training process of steps 210 to 230 again, the above process is repeated until the training completion condition is reached, wherein the training completion condition includes at least one of the following: the first channel network converges, the first channel network and the second channel network both converge, and the number of iterations reaches a number threshold. Illustratively, the sample image set contains N sample images, when the first random grouping is performed, N / 2 first sample image pairs can be obtained, and the first channel network and the second channel network are trained based on the sample images in each first sample image pair; if the training of the first channel network and the second channel network based on the N / 2 first sample image pairs is completed, the training completion condition is still not reached, then the second random grouping is performed on the N sample images to obtain N / 2 second sample image pairs, and the first channel network and the second channel network are trained based on the sample images in each second sample image pair, and the above process is repeated until the training completion condition is reached. In this process, when the random grouping is performed, there can be n*n different combinations of image sample pairs, thereby realizing the exponential increase of the number of learnable sample images.

[0077] In the embodiments of the present application, the channel network obtained after the training of the first channel network is used to construct an image classification model, and the image classification model is used to predict the type to which a target medical image belongs.

[0078] Illustratively, the channel network obtained after the training of the first channel network can be acquired as an image classification model, or the model can be reconstructed based on the structure and parameters of the channel network obtained after the training of the first channel network to obtain the image classification model. The process of training the first channel network and the second channel network can be performed by a terminal or a server. If the process is performed by the server, when the channel network obtained after the training of the first channel network is acquired, the server can deploy the channel network obtained after the training of the first channel network as an image classification model, or the structure and parameters of the channel network obtained after the training of the first channel network can be pushed to a deployment device, so that the deployment device constructs an image classification model based on the obtained structure and parameters of the first channel network. The deployment device can be implemented as a terminal or a server, or the image classification model can be constructed by relevant personnel based on the structure and parameters of the first channel network, and the image classification model can be deployed.

[0079] In summary, the image processing method for medical images provided in the embodiments of the present application trains the first channel network and the second channel network by using the first sample image and the second sample image, so that the prediction classification result obtained based on the second channel network can assist the training of the first channel network, thereby improving the classification accuracy of the image classification model obtained based on the trained first channel network.

[0080] Meanwhile, based on the network training of two sample images at the same time, the number of learnable sample images can be exponentially increased, thereby avoiding the overfitting phenomenon caused by the limited number of sample images in the small sample learning scenario, and improving the training effect of the model.

[0081] The application scenarios in the schemes described in the embodiments of the present application include but are not limited to the following scenarios:

[0082] 1) Assisting medical staff in esophageal cancer recurrence detection scenarios:

[0083] Esophageal cancer is a malignant tumor occurring in the esophagus and originating from esophageal epithelial cells. In clinical applications, the estimation of the recurrence of esophageal cancer will affect clinical decision-making, and different treatment methods will be taken for high-risk / low-risk populations. In clinical applications, the corresponding medical images can be acquired by a medical image acquisition device to determine the recurrence risk of esophageal cancer. To improve the prediction accuracy of the recurrence risk of esophageal cancer, the esophageal cancer medical images acquired by the medical image acquisition device can be input into the image classification model obtained based on the image processing method for medical images provided in the embodiments of the present application, and the prediction type result output by the image classification model for the esophageal cancer medical images can be obtained. The prediction type result is used to indicate the recurrence risk of esophageal cancer, such as no recurrence risk, low recurrence risk, medium recurrence risk, high recurrence risk, etc.

[0084] 2) Assist medical staff in medical image lesion judgment scenarios:

[0085] In the medical field, medical staff often judges the possibility of the presence of lesions in organs through medical images obtained by medical image acquisition devices, such as lesion examination of the stomach, lung tumor examination, brain tumor examination, etc. In the above scenarios, the image classification model corresponding to each of the above scenarios can be obtained by the image processing method for medical images provided in the present application, which is used to determine the possibility of the presence of lesions in organs, so that medical staff can reasonably allocate medical resources based on the possibility of the presence of lesions. Therefore, the image classification model obtained based on the image processing method for medical images provided in the present application can improve the accuracy of classifying medical images and further improve the accuracy of judging the possibility of the presence of lesions, thereby achieving reasonable allocation of medical resources.

[0086] The scheme related in the present application includes an image classification model generation stage and an image classification stage. Figure 3 is a framework diagram of image classification model generation and image classification according to an exemplary embodiment, as Figure 3 shown, in the image classification model generation stage, the image classification model generation device 310 trains the first channel network and the second channel network through the pre-set training sample data set (including different sample images and the type labels corresponding to each sample image), and obtains the trained first channel network; then, based on the obtained channel network after the first channel network is trained, an image classification model is generated. In the image processing classification stage, the image classification device 320 processes the input target medical image based on the image classification model, and obtains the image classification result of the target medical image, which is used to indicate the predicted type result corresponding to the target medical image, such as determining the recurrence risk of the specified disease corresponding to the target medical image.

[0087] Among them, the above-mentioned image classification model generation device 310 and image classification device 320 can be a computer device, such as a personal computer, a server, etc. Fixed computer device, or the computer device can also be a tablet computer, an electronic book reader, etc. Mobile computer device.

[0088] Optionally, the image classification model generation device 310 and the image classification device 320 can be the same device, or the image classification model generation device 310 and the image classification device 320 can also be different devices. Moreover, when the image classification model generation device 310 and the image classification device 320 are different devices, the image classification model generation device 310 and the image classification device 320 can be the same type of device, such as the image classification model generation device 310 and the image classification device 320 can both be servers; or the image classification model generation device 310 and the image classification device 320 can also be different types of devices, such as the image classification device 320 can be a personal computer or a terminal, and the image classification model generation device 310 can be a server, etc. The specific types of the image classification model generation device 310 and the image classification device 320 are not limited in the embodiments of the present application.

[0089] For example, the step of constructing the image classification model is executed by a computer device, Figure 4 A flowchart of an image processing method for medical images provided by an example embodiment of the present application is shown, which can be executed by a computing device, which can be implemented as Figure 1 The server shown or the server and the terminal, such as Figure 4 The image processing method for medical images includes the following steps:

[0090] Step 410, calling the first channel network to process the first sample image, obtaining a predicted type result corresponding to the first sample image and an image feature of the first sample image; the predicted type result is used to indicate the type to which the first sample image belongs.

[0091] In the embodiments of the present application, the first channel network includes m feature extraction layers, and the m feature extraction layers are connected layer by layer, and 2≤m, and m is a positive integer;

[0092] In response to the second target feature extraction layer being the first feature extraction layer of the first channel network, the input of the second target feature extraction layer is the first sample image;

[0093] In response to the second target feature extraction layer being the lth feature extraction layer of the first channel network, the input of the second target feature extraction layer includes the output result of the (l-1)th feature extraction layer; 2≤l≤m.

[0094] That is, for the first feature extraction layer of the first channel network, the corresponding input is the first sample image, and the input of the first feature extraction layer is the output result of the previous feature extraction layer.

[0095] In a possible implementation, to further reduce the overfitting phenomenon of small sample learning caused by a small number of samples, before the first channel network is invoked to process the first sample image and obtain the predicted type result corresponding to the first sample image, the method further includes:

[0096] performing data enhancement on the first sample image to obtain a data-enhanced first sample image;

[0097] Then, the first channel network is invoked to process the data-enhanced first sample image, and the predicted type result corresponding to the first sample image is obtained.

[0098] The data enhancement includes at least one of cropping, random rotation, and random translation.

[0099] The first sample image and the second sample image are three-dimensional images. Illustratively, the three-dimensional images can be positron emission tomography (PET) images or computed tomography (CT) images. Correspondingly, the data enhancement includes random three-dimensional rotation, random translation, and cropping. The random rotation refers to randomly rotating the three-dimensional image by 0 to 360 degrees in the x, y, and z directions. The random translation refers to randomly moving the three-dimensional image by different pixel points in the x, y, and z directions, for example, 0 to 15 different pixel points in the x, y, and z directions.

[0100] In step 420, the second channel network is invoked to process the image features of the second sample image and the first sample image, and a predicted classification result of the second sample image relative to the first sample image is obtained. The predicted classification result is used to indicate whether the second sample image and the first sample image are images of the same type.

[0101] In the embodiment of the present application, the second channel network includes n feature extraction layers, and the n feature extraction layers are connected layer by layer, where 2≤n≤m, and n is a positive integer.

[0102] In the embodiment of the present application, the image features of the first sample image can be intermediate features obtained by processing the first sample image through the first channel network, or are intermediate features obtained by processing the data-enhanced first sample image through the first channel network.

[0103] Since the first channel network comprises m feature extraction layers, the intermediate features of the first sample image can be obtained by processing through one feature extraction layer; therefore, when obtaining the predicted classification result of the second sample image compared with the first sample image through the second channel network, the image features of the first sample image can be input into the corresponding feature extraction layers of the second channel network in sequence, so that the second channel network can obtain the predicted classification result of the second sample image compared with the first sample image based on the extracted image features of the second sample image and the obtained image features of the first sample image.

[0104] That is, in response to the first target feature extraction layer being the first feature extraction layer of the second channel network, the input of the first target feature extraction layer is the second sample image;

[0105] In response to the first target feature extraction layer being the i th feature extraction layer of the second channel network, the input of the first target feature extraction layer comprises the output result of the i-1 th feature extraction layer of the second channel network and the output result of the i-1 th feature extraction layer of the first channel network; 2≤i≤n.

[0106] In a possible implementation, the second sample image is subjected to data enhancement to obtain a data-enhanced second sample image.

[0107] Then, the second channel network is called to process the data-enhanced second sample image and the image features of the first sample image, and obtain the predicted classification result of the second sample image compared with the first sample image.

[0108] For example, when m=n, Figure 5 a schematic diagram of the first channel network and the second channel network according to an example embodiment of the present application is shown, as Figure 5As shown, the backbone networks in the first channel network 510 and the second channel network 520 respectively include n feature extraction layers, which can implement a residual network (ResNet) in the embodiments of the present application, and the n feature extraction layers can be implemented as n residual blocks in the residual network. The first channel network 510 is a main channel, and the final output classification result is the predicted type result of the first sample image 511. The second channel network 520 is a collaborative channel, and the final output prediction classification result is used to indicate whether the first sample image 511 and the second sample image 521 are images of the same class. In the training stage, the first sample image and the second sample image correspond to medical sample images of esophageal cancer, and the labels of each medical sample image are esophageal cancer recurrence or esophageal cancer non-recurrence. For example, the first block of the first channel network 510 receives a data-enhanced first sample image input, and the first block of the second channel network receives a data-enhanced second sample image input. Starting from the second block, for the first channel network 510, the input of each block is the output of the last block of the first channel network 510, and for the second channel network, the input of each block is the sum of the output of the last block of the first channel network 510 and the output of the last block of the second channel network 520. Based on this, the first channel network and the second channel network can focus on learning different content, so that the first channel network focuses on predicting the type of the sample image, and the second channel network focuses on distinguishing the difference between different categories of images, and the difference is fed back to the first channel network during the training process to assist the training of the first channel network.

[0109] Figure 6 A schematic diagram of the backbone network according to an example embodiment of the present application is shown as follows. Figure 6 As shown, in the embodiments of the present application, the data-enhanced sample image 610 is input into the residual network, and based on the processing of at least two residual blocks in the residual network, the image features corresponding to the data-enhanced sample image are obtained. Figure 6 As shown, the at least two residual blocks can be implemented as at least two bottleneck layers.

[0110] In the training process of the image processing model, it mainly includes two stages of forward propagation and backward feedback. The forward propagation stage is shown as follows. Figure 5 As shown, the first sample image and the second sample image are respectively subjected to data enhancement processing, and the image features of the first sample image and the second sample image are extracted. Based on the image features of the first sample image and the second sample image, the prediction classification result and the prediction type result of the first sample image are obtained.

[0111] The backward feedback stage is a process of updating parameters of the first channel network and the second channel network according to the predicted classification result obtained in the forward propagation stage, the predicted type result of the first sample image, the type label corresponding to the first sample image, and the type label corresponding to the second sample image.

[0112] In step 430, parameters of the second channel network are updated based on a difference between the predicted classification result and a classification result label, the classification result label being determined based on a type label of the first sample image and a type label of the second sample image.

[0113] Since different sample images have different type labels, whether the sample images input into the first channel network and the second channel network respectively are images of the same type can be known based on the type labels of the sample images, that is, the classification result label is determined.

[0114] The process of updating parameters of the second channel network can be implemented as follows:

[0115] A function value of the first loss function is calculated based on a difference between the predicted classification result and the classification result label.

[0116] Parameters of the second channel network are updated based on the function value of the first loss function.

[0117] In step 440, parameters of the first channel network are updated based on a difference between the predicted type result and the type label of the first sample image, and a difference between the predicted classification result and the classification result label.

[0118] In the embodiments of the present application, parameters of the first channel network and the second channel network can be updated based on a difference between the predicted classification result and the classification result label, and parameters of the first channel network can be further updated based on a difference between the predicted type result and the type label of the first sample image.

[0119] The process of updating parameters of the first channel network can be implemented as follows:

[0120] A function value of the first loss function is calculated based on a difference between the predicted classification result and the classification result label.

[0121] A function value of the second loss function is calculated based on a difference between the predicted classification result and the type label of the first sample image.

[0122] Parameters of the first channel network are updated based on the function value of the first loss function and the function value of the second loss function.

[0123] In the embodiments of the present application, the first loss function and the second loss function can be implemented as one or more of a classification loss function, such as a cross-entropy loss function, a mean square error loss function, an exponential loss function, a negative log-likelihood loss, etc.

[0124] In the embodiments of the present application, when updating the parameters of the first channel network, the function values of the first loss function and the function values of the second loss function can be weighted and summed to obtain a weighted sum result.

[0125] Based on the weighted result, the parameters of the first channel network are updated.

[0126] Based on the first sample image and the second sample image in different pairs of sample images, the steps 410 to 440 are iteratively executed until a training completion condition is reached, and the trained first channel network and the trained second channel network are obtained.

[0127] Step 450: Based on the channel network obtained after the training of the first channel network is completed, an image classification model is constructed, which is used to predict the type to which a target medical image belongs.

[0128] The target medical image is a medical image input into the image classification model.

[0129] After obtaining the image classification model, the image classification model can be deployed on a deployment device, which can be implemented as a terminal or a server.

[0130] When applying the image classification model, the target medical image can be input into the image classification model to obtain a prediction type result of the target medical image output by the image classification model.

[0131] Alternatively, in a possible implementation manner, in order to enhance the robustness of the prediction result, random data augmentation can be performed on the target medical image to obtain x random data augmentation results corresponding to the target medical image.

[0132] The x random data augmentation results are sequentially input into the image classification model to obtain prediction type results corresponding to the x random data augmentation results, respectively.

[0133] Based on the prediction type results corresponding to the x random data augmentation results, respectively, a prediction type result of the target medical image is obtained.

[0134] Optionally, the x random data augmentation results are different images obtained by rotating, translating or cropping the target medical image.

[0135] Illustratively, the average value of the prediction type results corresponding to the x random data augmentation results respectively is obtained as the prediction type result of the target medical image, and the corresponding formula is:

[0136]

[0137] wherein p i represents the prediction type result corresponding to the i-th random data augmentation result in the x prediction type results, and p represents the prediction type result of the target medical image.

[0138] To sum up, the image processing method for medical images provided by the embodiments of the present application trains the first channel network and the second channel network by using the first sample image and the second sample image, so that the prediction classification result obtained based on the second channel network can assist the training of the first channel network, thereby improving the classification accuracy of the image classification model obtained based on the trained first channel network.

[0139] At the same time, based on the network training of two sample images at the same time, the number of learnable sample images can be exponentially increased, thereby avoiding the overfitting phenomenon caused by the limited number of samples in the small sample learning scenario, and improving the training effect of the model.

[0140] Figure 7 A block diagram of an image processing apparatus for medical images is shown, as shown in Figure 7 The image processing apparatus for medical images comprises:

[0141] The first processing module 710 is configured to call the first channel network, process the first sample image, and obtain the prediction type result corresponding to the first sample image and the image features of the first sample image. The prediction type result is used to indicate the type to which the first sample image belongs.

[0142] The second processing module 720 is configured to call the second channel network, process the second sample image and the image features of the first sample image, and obtain the prediction classification result of the second sample image compared with the first sample image. The prediction classification result is used to indicate whether the second sample image and the first sample image are images of the same type.

[0143] The network training module 730 is configured to train the first channel network and the second channel network based on the prediction type result, the prediction classification result, the type label of the first sample image, and the type label of the second sample.

[0144] The first channel network is trained to obtain a channel network, and the channel network is used to construct an image classification model, and the image classification model is used to predict a type of a target medical image.

[0145] In a possible implementation, the second channel network comprises n feature extraction layers connected layer by layer, and the first channel network comprises m feature extraction layers connected layer by layer, 2≤n≤m, and n and m are positive integers.

[0146] In response to the first target feature extraction layer being the first feature extraction layer of the second channel network, an input of the first target feature extraction layer is the second sample image.

[0147] In response to the first target feature extraction layer being the i th feature extraction layer of the second channel network, an input of the first target feature extraction layer comprises an output result of an (i-1) th feature extraction layer of the second channel network and an output result of an (i-1) th feature extraction layer of the first channel network, and 2≤i≤n.

[0148] In a possible implementation, in response to the second target feature extraction layer being the first feature extraction layer of the first channel network, an input of the second target feature extraction layer is the first sample image.

[0149] In response to the second target feature extraction layer being the l th feature extraction layer of the first channel network, an input of the second target feature extraction layer comprises an output result of an (l-1) th feature extraction layer, and 2≤l≤m.

[0150] In a possible implementation, the network training module 730 comprises:

[0151] The first parameter updating submodule is configured to perform parameter updating on the second channel network based on a difference between the predicted classification result and a classification result label, and the classification result label is a classification result determined based on a type label of the first sample image and a type label of the second sample image.

[0152] The second parameter updating submodule is configured to perform parameter updating on the first channel network based on a difference between the predicted type result and the type label of the first sample image, a difference between the predicted classification result and the classification result label.

[0153] In a possible implementation, the first parameter updating submodule is configured to calculate a function value of a first loss function based on the difference between the predicted classification result and the classification result label.

[0154] update parameters of the second channel network based on the function value of the first loss function.

[0155] In a possible implementation, the second parameter updating submodule is configured to calculate a function value of a first loss function based on a difference between the predicted classification result and a classification result label.

[0156] calculate a function value of a second loss function based on a difference between the predicted classification result and a type label of the first sample image.

[0157] update parameters of the first channel network based on the function value of the first loss function and the function value of the second loss function.

[0158] In a possible implementation, the second parameter updating submodule is configured to perform weighted summation on the function value of the first loss function and the function value of the second loss function to obtain a weighted summation result.

[0159] update parameters of the first channel network based on the weighted result.

[0160] In a possible implementation, the apparatus further includes:

[0161] a first data enhancement module configured to perform data enhancement on the first sample image to obtain a data-enhanced first sample image.

[0162] The first processing module 710 is configured to invoke the first channel network, process the data-enhanced first sample image, and obtain the predicted type result corresponding to the first sample image.

[0163] In a possible implementation, the apparatus further includes:

[0164] a second data enhancement module configured to perform data enhancement on the second sample image to obtain a data-enhanced second sample image.

[0165] The second processing module 720 is configured to invoke the second channel network, process the data-enhanced second sample image and the image features of the first sample image, and obtain the predicted classification result of the second sample image relative to the first sample image.

[0166] In a possible implementation, the data enhancement includes at least one of random rotation and random translation.

[0167] In a possible implementation, the apparatus further includes:

[0168] The first data enhancement module is configured to perform data enhancement on the target medical image to obtain x data enhancement results corresponding to the target medical image.

[0169] The first obtaining module is configured to sequentially input the x random enhancement results into the image classification model to obtain the prediction type results corresponding to the x random enhancement results, respectively.

[0170] The second obtaining module is configured to obtain the prediction type result of the target medical image based on the prediction type results corresponding to the x random enhancement results, respectively.

[0171] In summary, the image processing method for medical images provided by the embodiments of the present application trains the first channel network and the second channel network by using the first sample image and the second sample image, so that the prediction classification result obtained based on the second channel network can assist the training of the first channel network, thereby improving the classification accuracy of the image classification model obtained based on the trained first channel network.

[0172] Meanwhile, based on the network training of two sample images at the same time, the number of learnable sample images can be exponentially increased, thereby avoiding the overfitting phenomenon caused by the limited number of sample images in the small sample learning scenario, and improving the training effect of the model.

[0173] Figure 8 A structural block diagram of a computer device 800 is shown, which is an example of the computer device according to an embodiment of the present application. The computer device can be implemented as a server in the above-mentioned solutions of the present application. The computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including a random access memory (RAM) 802 and a read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the central processing unit 801. The computer device 800 further includes a mass storage device 806 for storing an operating system 809, application programs 810 and other program modules 811.

[0174] The mass storage device 806 is connected to the central processing unit 801 through a mass storage controller (not shown) connected to the system bus 805. The mass storage device 806 and its associated computer readable medium provide non-volatile storage for the computer device 800. That is, the mass storage device 806 can include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0175] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, Digital Versatile Disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that computer storage media does not limit to the above-mentioned several kinds. The system memory 804 and the mass storage device 806 mentioned above can be collectively referred to as memory.

[0176] According to various embodiments of the present disclosure, the computer device 800 can also run on a remote computer connected to a network such as the Internet. That is, the computer device 800 can be connected to a network 808 through a network interface unit 807 connected to the system bus 805, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 807.

[0177] The memory further includes at least one instruction, at least one program, a code set or an instruction set, which is stored in the memory, and the central processing unit 801 implements all or part of the steps of the image processing method for medical images shown in various embodiments by executing the at least one instruction, at least one program, code set or instruction set.

[0178] Figure 9 The structural block diagram of a computer device 900 provided by an exemplary embodiment of the present application is shown. The computer device 900 can be implemented as the terminal described above, such as a smart phone, a tablet computer, a notebook computer or a desktop computer. The computer device 900 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0179] Generally, the computer device 900 includes a processor 901 and a memory 902.

[0180] The processor 901 can include one or more processing cores, such as a 4-core processor, a 9-core processor, and the like. The processor 901 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array), and the like. The processor 901 can also include a main processor and a co-processor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the co-processor being a low-power consumption processor for processing data in a standby state. In some embodiments, the processor 901 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 901 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0181] The memory 902 can include one or more computer-readable storage media that can be non-transitory. The memory 902 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction for being executed by the processor 901 to implement all or part of the steps in the image processing method for medical images provided by the method embodiments in the present application.

[0182] In some embodiments, the computer device 900 can also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface 903 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 903 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 904, a display screen 905, a camera component 906, an audio circuit 907, a positioning component 908, and a power supply 909.

[0183] The peripheral interface 903 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902 and the peripheral interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902 and the peripheral interface 903 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0184] In some embodiments, the computer device 900 further comprises one or more sensors 910. The one or more sensors 910 include, but are not limited to, an acceleration sensor 911, a gyroscope sensor 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915 and a proximity sensor 916.

[0185] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the computer device 900, and the computer device 900 can include more or fewer components than those shown in the figure, or combine certain components, or adopt a different arrangement of components. Figure 9

[0186] In an exemplary embodiment, a computer readable storage medium is also provided for storing at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement all or part of the steps of the above-mentioned image processing method for medical images. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0187] In an exemplary embodiment, a computer program product or computer program is also provided, which comprises computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform all or part of the steps of the above-mentioned image processing method for medical images. Figure 2 or Figure 4 all or part of the steps of the method shown in any of the embodiments.

[0188] ​Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0189] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various changes in shape, size and arrangements of parts can be made without departing from the scope of the application. The scope of the application is only limited by the appended claims.

Claims

1. An image processing method for medical images, characterized in that, The method comprises: calling a first channel network to process a first sample image, to obtain a predicted type result corresponding to the first sample image and an image feature of the first sample image; the predicted type result is used to indicate a type to which the first sample image belongs; calling a second channel network to process a second sample image and the image feature of the first sample image, to obtain a predicted classification result of the second sample image relative to the first sample image; the predicted classification result is used to indicate whether the second sample image and the first sample image are images of the same type; based on a difference between the predicted classification result and a classification result label, updating parameters of the second channel network, the classification result label being used to indicate a classification result determined based on a type label of the first sample image and a type label of the second sample image; based on a difference between the predicted type result and the type label of the first sample image, and a difference between the predicted classification result and the classification result label, updating parameters of the first channel network; wherein a channel network obtained after the first channel network is trained is used to construct an image classification model, and the image classification model is used to predict a type to which a target medical image belongs.

2. The method of claim 1, wherein, The second channel network comprises n feature extraction layers, and the n feature extraction layers are connected layer by layer; the first channel network comprises m feature extraction layers, and the m feature extraction layers are connected layer by layer; 2≤n≤m, and n and m are positive integers; in response to a first target feature extraction layer being a first feature extraction layer of the second channel network, an input of the first target feature extraction layer is the second sample image; in response to the first target feature extraction layer being an i-th feature extraction layer of the second channel network, an input of the first target feature extraction layer comprises an output result of an (i-1)-th feature extraction layer of the second channel network and an output result of an (i-1)-th feature extraction layer of the first channel network; 2≤i≤n. in response to a second target feature extraction layer being a first feature extraction layer of the first channel network, an input of the second target feature extraction layer is the first sample image; 3. The method of claim 2, wherein, in response to the second target feature extraction layer being an l-th feature extraction layer of the first channel network, an input of the second target feature extraction layer comprises an output result of an (l-1)-th feature extraction layer; 2≤l≤m. The updating of the parameters of the second channel network based on the difference between the predicted classification result and the classification result label comprises:

4. The method of claim 1, wherein, calculating a function value of a first loss function based on the difference between the predicted classification result and the classification result label; updating the parameters of the second channel network based on the function value of the first loss function. The updating of the parameters of the first channel network based on the difference between the predicted type result and the type label of the first sample image, and the difference between the predicted classification result and the classification result label comprises:

5. The method of claim 1, wherein, calculating a function value of a first loss function based on the difference between the predicted classification result and the classification result label; ​ compute a function value of a second loss function based on a difference between the predicted classification result and a type label of the first sample image; perform parameter updating on the first channel network based on the function value of the first loss function and the function value of the second loss function.

6. The method of claim 5, wherein, The parameter updating on the first channel network based on the function value of the first loss function and the function value of the second loss function comprises: perform weighted summation on the function value of the first loss function and the function value of the second loss function to obtain a weighted summation result; perform parameter updating on the first channel network based on the weighted summation result.

7. The method of claim 1, wherein, Before the calling of the first channel network and the processing of the first sample image to obtain the predicted type result corresponding to the first sample image, the method further comprises: perform data enhancement on the first sample image to obtain the first sample image after data enhancement; The calling of the first channel network and the processing of the first sample image to obtain the first classification result corresponding to the first sample image comprises: calling the first channel network to process the first sample image after data enhancement to obtain the predicted type result corresponding to the first sample image.

8. The method of claim 1, wherein, Before the calling of the second channel network and the processing of the second sample image and the image feature of the first sample image to obtain the predicted classification result of the second sample image relative to the first sample image, the method further comprises: perform data enhancement on the second sample image to obtain the second sample image after data enhancement; The calling of the second channel network and the processing of the second sample image and the image feature of the first sample image to obtain the predicted classification result of the second sample image relative to the first sample image comprises: calling the second channel network to process the second sample image after data enhancement and the image feature of the first sample image to obtain the predicted classification result of the second sample image relative to the first sample image.

9. The method according to any of claims 7 or 8, characterized in that, The data enhancement comprises at least one of cropping, random rotation and random translation.

10. The method of claim 1, wherein, After the acquisition of the image classification model, the method further comprises: perform random data enhancement on the target medical image to obtain x random data enhancement results corresponding to the target medical image; input the x random data enhancement results into the image classification model in sequence to obtain the predicted type result corresponding to each of the x random data enhancement results; acquire the predicted type result of the target medical image based on the predicted type result corresponding to each of the x random data enhancement results.

11. The method according to any of claims 1 to 8, 10, characterized in that, The first sample image and the second sample image are three-dimensional images.

12. An image processing apparatus for medical images, characterized by The apparatus comprises: a first processing module configured to call a first channel network to process a first sample image to obtain a predicted type result corresponding to the first sample image and an image feature of the first sample image; the predicted type result is used to indicate a type to which the first sample image belongs; The second processing module is configured to call a second channel network, process the second sample image and image features of the first sample image, and obtain a predicted classification result of the second sample image compared with the first sample image; the predicted classification result is used to indicate whether the second sample image and the first sample image are images of the same type; The network training module is configured to update parameters of the second channel network based on a difference between the predicted classification result and a classification result label, the classification result label being used to indicate a classification result determined based on a type label of the first sample image and a type label of the second sample image; and update parameters of the first channel network based on a difference between the predicted type result and the type label of the first sample image, the difference between the predicted classification result and the classification result label. The channel network obtained after the first channel network is trained is used to construct an image classification model, and the image classification model is used to predict a type to which a target medical image belongs.

13. A computer device, comprising: The computer device includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the image processing method for medical images according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to implement the image processing method for medical images according to any one of claims 1 to 11.

15. A computer program product, characterised in that, The computer program product includes computer instructions, which are read and executed by the processor of the computer device to implement the image processing method for medical images according to any one of claims 1 to 11.

Images