Medical image classification method, system and electronic equipment based on deep reinforcement learning

Abstract

This application discloses a medical image classification method, system, and electronic device based on deep reinforcement learning. The binary imbalanced classification problem of medical images is expressed as a sequential decision problem. It includes two models: an imbalanced classification model based on DDQN (DDQNbic) and an imbalanced classification model based on Dueling DQN (D_DQNbic). The system for image classification trains the training data after selecting the classification model. Users can choose according to their needs, or classify through different models and use them for result comparison. When the user selects a model, it will process the class-imbalanced images based on deep reinforcement learning technology, and ultimately successfully classify the images with higher accuracy. At the same time, this model directly classifies medical images with imbalanced categories without performing additional operations such as balancing samples, saving classification time and reducing operating costs.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and specifically relates to a medical image classification method, system and electronic equipment based on deep reinforcement learning. Background Art

[0002] Medical image classification is one of the challenges in image analysis. It requires not only high-precision identification of targets but also the location of disease, which can assist in medical diagnosis. Machine learning is widely used in image classification, enhancing the classification process through trial-and-error learning. Deep learning, in particular, has made significant progress. Zhang et al. proposed a collaborative deep learning model, in which each pair of deep convolutional neural networks learns from each other and discriminates features from multiple images to achieve classification tasks. Yadav et al. studied the use of convolutional neural networks to classify pneumonia images and compared the performance of three existing techniques. However, these techniques are all designed for images with balanced categories. Because medical images are difficult to collect, medical image datasets are more prone to category imbalance.

[0003] Imbalanced data classification issues are primarily addressed from two perspectives: data and algorithms. Taking binary classification as an example, Galdran et al. proposed a new sampling mechanism to create more balanced training sets. To address data imbalance, Xiao et al. proposed a new class-specific extreme learning machine (ELM) to rebalance the sample distribution. This machine can address misclassification and mitigate the negative impact of class imbalance. However, rebalancing samples is time-consuming, especially when the sample size is large. Huynh et al. applied an adaptive hybrid consistency loss (A) to semi-supervised learning to offset class imbalance and achieve correct classification. Lin et al. designed a binary imbalanced classification model based on the DQN algorithm, demonstrating that this model outperformed other methods. Stember et al. used the DQN algorithm to train brain tumor image classification, demonstrating that DRL can be effectively trained even on very small datasets. However, with technological advancements, these models are gradually falling behind.

[0004] However, the classification methods primarily employed in these studies aim to balance samples with imbalanced categories. This involves adding a balancing module, which increases computational time and costs. Especially when the sample size is large, the time required to balance samples can even exceed the time required to classify the images. Some research directly classifies samples with imbalanced categories. However, with the continuous advancement and development of machine learning technologies, existing methods are becoming increasingly obsolete. Furthermore, existing solutions often rely on a single model. When users have diverse needs and desire to compare multiple models, existing technologies offer no solution. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a medical image classification method, system and electronic device based on deep reinforcement learning, which can solve the technical problems involved in the background technology.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] A medical image classification method based on deep reinforcement learning, including:

[0008] Step S1: providing training samples;

[0009] Step S2: Select a model through the agent, where the model includes a first imbalanced classification model based on DDQN and a second imbalanced classification model based on Dueling DQN. If the first imbalanced classification model is selected, step S3 is executed; if the second imbalanced classification model is selected, step S5 is executed.

[0010] Step S3: The first imbalanced classification model is used to select an action of a training sample through a current value network, where the action includes a minority class and a majority class;

[0011] Step S4: After selecting an action, the first imbalanced classification model evaluates the selected action through the target value network and finds the reward of the action from the target value network;

[0012] Step S5: The second imbalanced classification model uses a convolutional network to extract image features of the training samples, obtains corresponding feature vectors, and then uses a competitive network to output the feature vectors to two fully connected layer branches, corresponding to the state value and advantage value respectively, to determine whether the training samples meet the state value or advantage value, and give corresponding rewards;

[0013] Step S6: The agent determines whether the classification of the training sample is accurate based on the level of the reward.

[0014] Optionally, in step S1, the training samples are medical images to be classified.

[0015] Optionally, in step S5, the state value represents the value of the static state environment itself; the advantage value represents the additional value brought by selecting a certain action.

[0016] Optionally, the rewards include:

[0017] If the correct classification action is performed, a positive reward will be obtained; otherwise, a negative reward will be given.

[0018] Optionally, the reward for training examples from the minority class is higher than the reward for training examples from the majority class.

[0019] The present application provides a medical image classification system based on deep reinforcement learning, including:

[0020] A training sample providing module is used to provide training samples;

[0021] a model selection module, configured to select a model through an agent, the model comprising a first imbalanced classification model based on DDQN and a second imbalanced classification model based on Dueling DQN;

[0022] The classification accuracy judgment module is used to judge whether the classification of training samples is accurate based on the level of rewards.

[0023] Optionally, the first imbalanced classification model is used to select actions of training samples through a current value network, where the actions include minority classes and majority classes; after selecting the action, the first imbalanced classification model evaluates the selected action through a target value network and finds the reward of the action from the target value network.

[0024] Optionally, the second unbalanced classification model uses a convolutional network to extract image features of training samples, obtains corresponding feature vectors, and then uses a competitive network to output the feature vectors to two fully connected layer branches, corresponding to state value and advantage value respectively, to determine whether the training samples meet the state value or advantage value, and give corresponding rewards.

[0025] An embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the steps of the described method when executed by the processor.

[0026] An embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described above are implemented.

[0027] The beneficial effects of this application are as follows:

[0028] 1. This application integrates deep reinforcement learning technology, which pays greater attention to minority classes by allocating higher rewards. Its learning mechanism can correctly classify medical images with imbalanced categories.

[0029] 2. The first unbalanced classification model adopted in this application enables action selection and strategy evaluation to be implemented using different value functions, which effectively reduces the risk of overestimation of the Q value.

[0030] 3. This application adopts two different imbalanced classification models for classification and comparison of results. When the user selects a model, it will process the imbalanced images based on deep reinforcement learning technology, and finally successfully classify the images with higher accuracy. At the same time, this model directly classifies the imbalanced medical images without performing additional operations such as balancing samples, saving classification time and reducing operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of a medical image classification method based on deep reinforcement learning provided in an embodiment of the present application;

[0032] Figure 2 This is a structural block diagram of a medical image classification system based on deep reinforcement learning provided in an embodiment of the present application;

[0033] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0036] The following describes in detail the medical image classification method based on deep reinforcement learning provided by the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0037] See Figure 1 , is a flow chart of a medical image classification method based on deep reinforcement learning provided in an embodiment of the present application, such as Figure 1 As shown, the above-mentioned medical image classification method based on deep reinforcement learning includes:

[0038] Step S1: providing a training sample, wherein the training sample is a medical image to be classified;

[0039] Step S2: Select a model through the agent. The model includes a first imbalanced classification model (DDQNbic) based on DDQN and a second imbalanced classification model (D_DQNbic) based on Dueling DQN. If the first imbalanced classification model is selected, step S3 is executed; if the second imbalanced classification model is selected, step S5 is executed.

[0040] Step S3: The first imbalanced classification model is used to select an action of training samples through the current value network, wherein the action includes a minority class and a majority class. In the specific embodiment of the present application, the minority class is represented by 0 and the majority class is represented by 1;

[0041] Step S4: After selecting an action, the first imbalanced classification model evaluates the selected action through a target value network and finds a reward for the action from the target value network. Specifically, the reward includes: a positive reward for performing a correct classification action; otherwise, a negative reward.

[0042] Step S5: The second imbalanced classification model uses a convolutional network to extract image features of the training samples, obtains corresponding feature vectors, and then uses a competitive network to output the feature vectors to two fully connected layer branches, corresponding to the state value and advantage value, respectively. The model determines whether the training sample meets the state value or advantage value, and gives a corresponding reward. Specifically, the state value represents the value of the static state environment itself; the advantage value represents the additional value brought by selecting a certain action.

[0043] It should be noted that the reward mechanism adopted by this application is that if the correct classification action is performed, a positive reward will be obtained; otherwise, a negative reward will be given. In order to obtain as much reward as possible in the Markov decision process, the samples will be identified as correctly as possible. It is worth noting that in order to identify as many minority class samples as possible, the reward for the minority class is higher than the reward for the majority class. This application proposes an imbalanced classification model (D_DQNimc) based on Dueling DQN for classifying binary imbalanced images. In the model of this application, the dueling architecture selected is 'avg', which uses an average method to calculate Q(s,a) from the state value function and the action advantage function.

[0044] Step S6: The agent determines whether the classification of the training sample is accurate based on the level of the reward.

[0045] In step S2, the first imbalanced classification model includes:

[0046] Assuming that the reward R obtained at each time step is multiplied by a discount factor r, the sum of the rewards from t to the end of T episodes is defined as:

[0047]

[0048]

[0049] Execute action a from the current state s and follow the strategy Π until the end of the episode, expressed as the state action value function , the cumulative reward obtained by the agent in this process is:

[0050]

[0051] Optimal strategy The expected return of is greater than or equal to the expected return of all other strategies. It has more than one, but shares one Q. Then the optimal action-state value function is:

[0052]

[0053] According to the Bellman optimality equation, we have:

[0054] .

[0055] It should be noted that the strategy Π is a mapping from state s to action a in deep reinforcement learning.

[0056] Traditional DQN algorithms often overestimate the Q-values of actions, and the estimation error increases with the number of actions. If the overestimation is uneven, the Q-value of a suboptimal action can exceed the Q-value of the optimal action, making it impossible to find the optimal policy. DDQN uses two different parameter networks: the current value network and the target value network. The current value network is used to select actions, while the target value network is used to evaluate actions. The target Q-value of the action is obtained from the target value network. This allows action selection and policy evaluation to be implemented using different value functions, effectively reducing the risk of Q-value overestimation. The final loss equation to be learned is:

[0057]

[0058] In this application, actions are the categories of training samples, and the agent performing actions means that the agent predicts the category of the training samples. For the unbalanced binary classification problem to be solved in this application, the minority class is represented by 0 and the majority class is represented by 1. The agent is in state Execute an action Reward received. In this framework, rewards are used to measure whether the agent successfully classifies a sample. If the agent correctly performs the classification action, it receives a positive reward; otherwise, it receives a negative reward. Specifically, when a sample belongs to the minority class, if the agent performs the action that matches the sample label, i.e., correctly classified, the reward is 1; otherwise, the reward is 0. Similarly, when a sample belongs to the majority class, the agent's reward for a correct classification is Cf / Cm, and for an error is -Cf / Cm. Cf represents a sample from the minority class, and Cm represents a sample from the majority class.

[0059] See also Figure 2 As shown, an embodiment of the present application provides a medical image classification system based on deep reinforcement learning, including: a training sample providing module 1, a model selection module 2 and a classification accuracy judgment module 3.

[0060] The training sample extraction module 1 is used to provide training samples.

[0061] The model selection module 2 is used to select a model through an agent, and the model includes a first imbalanced classification model based on DDQN and a second imbalanced classification model based on Dueling DQN.

[0062] Specifically, the first imbalanced classification model is used to select actions of training samples through a current value network, where the actions include minority classes and majority classes. After selecting the action, the first imbalanced classification model evaluates the selected action through a target value network and finds the reward of the action from the target value network.

[0063] Optionally, the second unbalanced classification model uses a convolutional network to extract image features of training samples, obtains corresponding feature vectors, and then uses a competitive network to output the feature vectors to two fully connected layer branches, corresponding to state value and advantage value respectively, to determine whether the training samples meet the state value or advantage value, and give corresponding rewards.

[0064] The classification accuracy judgment module 3 is used to judge whether the classification of the training sample is accurate according to the level of the reward.

[0065] The system in the embodiments of the present application may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), an ATM, or an kiosks, etc., without any specific limitation in the embodiments of the present application.

[0066] Optional, such as Figure 3 As shown, an embodiment of the present application also provides an electronic device 300, including a processor 301, a memory 302, and a program or instruction stored in the memory 302 and executable on the processor 301. When the program or instruction is executed by the processor 301, each process of the above-mentioned embodiment of the medical image classification method based on deep reinforcement learning is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0067] It should be noted that the first electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device mentioned above.

[0068] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned embodiment of the medical image classification method based on deep reinforcement learning are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0069] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0070] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the medical image classification method based on deep reinforcement learning, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0071] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0072] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or system comprising the element. In addition, it should be noted that the scope of the methods and systems in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0073] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0074] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A medical image classification method based on deep reinforcement learning, characterized in that: include: Step S1: providing training samples; Step S2: Select a model through the agent, where the model includes a first imbalanced classification model based on DDQN and a second imbalanced classification model based on Dueling DQN. If the first imbalanced classification model is selected, step S3 is executed; if the second imbalanced classification model is selected, step S5 is executed. Step S3: The first imbalanced classification model is used to select an action of a training sample through a current value network, where the action includes a minority class and a majority class; Step S4: After selecting an action, the first imbalanced classification model evaluates the selected action through the target value network and finds the reward of the action from the target value network; Step S5: The second imbalanced classification model uses a convolutional network to extract image features of the training samples, obtains corresponding feature vectors, and then uses a competitive network to output the feature vectors to two fully connected layer branches, corresponding to the state value and advantage value respectively, to determine whether the training samples meet the state value or advantage value, and give corresponding rewards; Step S6: The agent determines whether the classification of the training sample is accurate based on the level of the reward.

2. The method according to claim 1, characterized in that In step S1, the training samples are medical images to be classified.

3. The method according to claim 1, characterized in that In step S5, the state value represents the value of the static state environment itself; the advantage value represents the additional value brought by selecting a certain action.

4. The method according to claim 1, wherein The rewards include: If the correct classification action is performed, a positive reward will be obtained; otherwise, a negative reward will be given.

5. The method according to claim 1, wherein The reward for training examples from the minority class is higher than the reward for training examples from the majority class.

6. A medical image classification system based on deep reinforcement learning for executing the method according to any one of claims 1 to 5, characterized in that: include: A training sample providing module is used to provide training samples; a model selection module, configured to select a model through an agent, the model comprising a first imbalanced classification model based on DDQN and a second imbalanced classification model based on Dueling DQN; The classification accuracy judgment module is used to judge whether the classification of training samples is accurate based on the level of rewards.

7. The system according to claim 6, characterized in that The first imbalanced classification model is used to select actions of training samples through a current value network, where the actions include a minority class and a majority class. After selecting the action, the first imbalanced classification model evaluates the selected action through a target value network and finds the reward of the action from the target value network.

8. The system according to claim 7, characterized in that The second imbalanced classification model uses a convolutional network to extract image features of training samples, obtains corresponding feature vectors, and then uses a competitive network to output the feature vectors to two fully connected layer branches, corresponding to the state value and advantage value respectively, to determine whether the training samples meet the state value or advantage value, and give corresponding rewards.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the medical image classification method based on deep reinforcement learning are implemented as described in any one of claims 1 to 5.

Images