Pathological image segmentation method based on multi-agent deep reinforcement learning
By treating each pixel in a pathological image as an agent, and utilizing multi-agent deep reinforcement learning and local refinement models for collaborative classification, the problems of limited data volume, high resolution, and difficulty in noise processing in pathological image segmentation are solved, achieving higher segmentation accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHWEST PETROLEUM UNIV
- Filing Date
- 2023-08-31
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies for pathological image segmentation suffer from problems such as limited data volume, high resolution, blurred boundaries, complex gradients, and difficulty in noise processing, resulting in low segmentation accuracy.
Each pixel is treated as an agent, and multi-agent deep reinforcement learning is used for collaboration. Pixel classification is performed through deep reinforcement learning and dominant action commenting algorithms, and fine segmentation is performed by combining local refinement models. Graph convolutional networks are used to extract pixel relationships and perform weighted adjustments.
It improves the accuracy and efficiency of pathological image segmentation, better handles image noise and boundary blurring, and achieves higher segmentation precision.
Smart Images
Figure CN117115182B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a pathological image segmentation method based on multi-agent deep reinforcement learning. Background Technology
[0002] In the field of image processing technology, pathological image segmentation is one of the important steps in the automated diagnosis, treatment and prognosis of diseases such as cancer. At present, the segmentation of pathological images still faces the following challenges: (1) the amount of data is small and pathological images with manual annotation by experts are scarce; (2) pathological images are large in size and have high resolution, which places certain demands on the processing speed of the model; (3) pathological images have blurred boundaries and complex gradients, which place high demands on the segmentation accuracy of the algorithm; (4) there are uncertainties in pathological images, such as image noise and annotation errors, and how to effectively deal with such uncertainties is a challenge. Summary of the Invention
[0003] The purpose of this invention is to improve upon the shortcomings of existing technologies and provide a pathological image segmentation method based on multi-agent deep reinforcement learning.
[0004] To address the technical problem of inaccurate pathological image segmentation in this application, the embodiments of the present invention provide the following technical solutions:
[0005] A pathological image segmentation method based on multi-agent deep reinforcement learning includes the following steps:
[0006] Step 1: Treat each pixel as an intelligent agent and collect the state dataset and similarity information of each pixel;
[0007] Step 2: Use deep reinforcement learning to estimate the probability that the pixel corresponding to the agent belongs to the segmentation target, and classify the pixel according to the estimated probability to determine whether the pixel belongs to the segmentation target.
[0008] When using deep reinforcement learning to estimate the probability that a pixel belongs to a segmentation target, the state dataset of the pixel and the similarity based on the pixel are used as inputs to the dominant action commenting algorithm. This allows the agent to better predict the probability that a pixel belongs to a segmentation target by using the states of neighboring pixels. Based on the states and similarities of neighboring pixels, a weight is assigned to each neighboring pixel, and these weights are used to adjust the prediction results, thereby improving the accuracy of segmentation.
[0009] Step 3: Repeat step 2 until all pixels in the image have undergone the above operation, and finally obtain the coarse segmentation result of the image;
[0010] Step 4: Within the coarsely segmented target, a multi-agent deep reinforcement learning algorithm is used to re-determine two pixels as new agents. These agents quickly find the pixel coordinates that need to be refined based on the known coarse segmentation results, and use a local refinement model to process the pixels that need to be refined, ultimately obtaining a refined segmentation target.
[0011] In the above scheme, each pixel is treated as an agent. Deep reinforcement learning is used to predict the probability that the pixel belongs to the segmentation target, which is the abnormal tissue or lesion area of a disease in a medical pathology image. The pixel is classified according to the probability, and the prediction result is adjusted by the dominant action review algorithm to improve the accuracy of segmentation. All pixels perform the above operation to complete the classification of all pixels in the image, and finally a coarse segmentation result is obtained. Based on the coarse segmentation result, two pixels are constructed as new agents. The new agents form a sliding window on the coarse segmentation result to find the coordinates of pixels that need to be further classified and perform further classification on these pixels, finally obtaining a more accurate fine segmentation target. Compared with traditional techniques, it can obtain an accurate segmentation target.
[0012] Furthermore, in step 1, the step of treating each pixel as an intelligent agent includes:
[0013] For a medical pathology image x = (x1, ..., xn) containing N pixels... i ,...,x N ), i = 1, 2, ..., N, each pixel x i Treat it as a single intelligent agent and share policies with other intelligent agents. in, Indicates an action, Indicates state;
[0014] By concatenating the pixel values x of the image i The previous iteration step on pixel value x i Estimated probability The segmentation target predicted in the previous iteration step Define state
[0015] action The set A = {A k}, k = 1, 2, ..., K, K contains K actions.
[0016] In the above scheme, each agent can communicate with other agents sharing the same strategy to exchange relevant information and further classify pixels based on the exchanged information. Thus, each agent observes the state of its neighbors through communication, enabling collaboration during pixel classification. Using multiple cooperating agents allows them to calculate more accurate probability estimates, resulting in more accurate pixel classification.
[0017] Furthermore, in step 1, the pixel similarity information includes nearest neighbors in feature space, color space, and distance space. The feature space includes manually created features or features extracted by neural networks; the color space includes RGB space and HSV space; and the distance space includes Euclidean distance or geodesic distance.
[0018] In the above scheme, by combining this similarity information, the agent can understand the relationship between pixels in the image, thereby improving the accuracy of image segmentation.
[0019] Furthermore, step 2, which involves estimating the agent using deep reinforcement learning and predicting the probability that the pixel corresponding to the agent belongs to the segmentation target, includes:
[0020] Execution strategy using the superior action commenting algorithm network. State and actions Perform mapping to obtain the pixel x i Estimated probability
[0021] The agent adjusts the reward based on the probability estimated at each iteration step. When the probability estimated in this iteration step Probability compared to the previous iteration step When there is an improvement, a reward will be given. It is a positive value; when the probability estimated in this iteration step is... Probability compared to the previous iteration step When there is a decrease, the reward It is a negative value.
[0022] Furthermore, the reward The difference in cross-entropy loss:
[0023]
[0024]
[0025]
[0026] in, This represents the cross-entropy loss at iteration step t. y represents the cross-entropy loss at iteration step t+1; i Represents pixel x i The truth value label.
[0027] Furthermore, for pixel x i After T iterations, the total reward is:
[0028]
[0029] Where, γ t-1 This represents the discount factor for iteration step t-1, ranging from 0 to 1.
[0030] Furthermore, in step 2, the dominant action commenting algorithm includes a graph convolutional network, a policy head, and a value head; the graph convolutional network extracts information about the relationships between pixels based on pixel similarity; the policy head is used to predict the probability that a pixel belongs to the segmentation target. Based on the state and similarity of neighboring pixels, a weight is assigned to each neighboring pixel, and then these weights are used to adjust the prediction results. A weighted adjustment is performed; the value head is used to estimate the current state. Value;
[0031] The strategy head includes a first convolutional block, a second convolutional block, a third convolutional block, a first upsampling layer, and a softmax layer connected in sequence; the value head includes a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, and a second upsampling layer connected in sequence.
[0032] Each convolutional block consists of a convolutional layer, a non-linear activation layer, and a batch normalization layer connected in sequence;
[0033] Each upsampling layer consists of either a bilinear upsampling layer or a transposed convolutional layer.
[0034] Furthermore, using θ g θ p θ v Let θ represent the model parameters of the graph convolutional network, policy head, and value head, respectively. During training, θ p θ v The gradient is calculated as follows:
[0035]
[0036]
[0037] in, Represents the differential operator; It is the probability that the i-th pixel at time step t, i.e., the i-th agent, chooses a in state s; Represents the logarithmic probability value; γ k-t This represents the discount factor from time step t to time step k; V represents the reward obtained by pixel i at time step k; i (s (t) ) represents the state value function of pixel i.
[0038] Furthermore, in step 4, within the coarsely segmented target, a multi-agent deep reinforcement learning algorithm is used to re-determine two pixels as new agents. These agents quickly search for the pixel coordinates that need to be refined based on the known coarse segmentation results. This step includes:
[0039] Two agents, Agent 1 and Agent 2, are constructed using a multi-agent deep reinforcement learning algorithm. The two agents pay attention to each other's state and actions. The window coordinates that need to be refined are found on the coarse segmentation results known in step 3.
[0040] The action space available to each agent consists of an M+1 dimensional vector. The first M dimensions of the vector represent the step size of the window the agent can move through, and the last dimension represents the sequence in which the agent chooses to stop. Therefore, the mathematical expressions for the action spaces of Agent 1 and Agent 2 are:
[0041] Action1 = {W 10 W 11 W 12 ,...,W 1(M-1) ,S1}
[0042] Action2 = {W 20 W 21 W 22 ,...,W 2(M-1) ,S2}
[0043] Where Action1 is the action space of agent one, S1 is the sequence for agent one to choose to stop, Action2 is the action space of agent two, S2 is the sequence for agent two to choose to stop, and W... ji Let i represent the different step sizes of the window that agent j can move, where j = 1, 2, i = 0, 1, ..., (M-1);
[0044] N y This represents the number of sliding windows the agent makes along the y-axis of the image. The window number i and its corresponding coordinates are:
[0045]
[0046] y = (i-1) mod N y
[0047] Where x and y represent the horizontal and vertical coordinates of the window.
[0048] Furthermore, in step 4, the step of re-identifying two pixels as new agents using a multi-agent deep reinforcement learning algorithm, and these agents quickly finding the pixel coordinates that need to be refined based on the known coarse segmentation results, also includes:
[0049] The policy function π(a|s) is used to define the policy adopted by the agent. The mathematical expression of the policy function is as follows:
[0050] π(a|s)=P(A t =a|S t =s)
[0051] When agent 1 takes action a1 in state s, the expected reward is:
[0052]
[0053] The expected reward r1(s,a1) represents the immediate reward of agent 1 in state s given agent 2's policy π2(a2|s) and state transition probability p(s`|s,a1,a2). In this way, agent 1 can consider the impact of agent 2's policy and state transition probability on its own policy and behavior, thereby achieving better cooperation and collaboration.
[0054] Given all possible actions a2 of agent 2, the probability that the environment transitions from state s to state s' after agent 1 takes action a1 is:
[0055]
[0056] The above formula calculates the probability of state transition given agent 1's action a1 and agent 2's policy π2(a2|s) by weighted summation over all possible actions of agent 2. In this way, agent 1 considers the influence of agent 2's policy on the state transition probability, thus more accurately predicting state transitions and making corresponding decisions.
[0057] The gradient of the expected return of agent j is:
[0058]
[0059] Wherein, J(θ) j The policy performance of agent j is represented by the expected reward, which is related to the policy parameters θ of agent j. j The function; logπ j (a j |s j ) indicates that in state s jBelow, agent j selects action a. j The logarithmic probability; This represents the policy π of a given agent j. j Given other agent policies, the state x and the values of actions a1 and a2 chosen by Agent 1 and Agent 2 are function functions; where state x = (o1, o2), o N (N∈[1,2]) represents the observations of agent j (j=1,2) at the current time step. The observations contain all the information acquired by agent j in the current state. This information includes, but is not limited to: the current state, the states of other agents, the actions of other agents, rewards, etc. The agent makes decisions and selects actions based on the observations in order to optimize its own strategy.
[0060] Furthermore, step 4, which involves processing the pixels that need to be refined and classified using a local refinement model to ultimately obtain a refined segmentation target, includes:
[0061] The environment in which the two agents exist can be viewed as a tuple as shown below:
[0062] Env = {Image, L Global ,P Global Pos x,y ,P Local ,L Local}
[0063] Where Image is the original medical pathology image, and L Global P is the original label given by the experts. Global Pos is the probability of segmenting the current image. x,y P selects the coordinates of the region for the agent. Local L is the segmentation probability for selecting a region for the agent. Local The label results for selecting regions for the agent. Among them, Image and P... Global The state is visible to the agent, and the label data L (L includes L) Global and L Local Pos is a hidden state that is not visible to the agent. x,y P Local This is the output of the intelligent agent.
[0064] During the interaction between the agent and the environment, the reward value fed back by the environment is R. t :
[0065]
[0066] R _j(t) =P double +D(x j(t) ,x i(t))
[0067] Among them, R t The reward value is the value obtained at time step t; the Dice coefficient is used to measure the segmentation probability image P at time step t. t Similarity between the data and the label data L; R _j(t) Let P be the individual reward obtained by agent j at time step t, where j = 1, 2, and P be the segmentation target estimation probability P of the pixel when the pixel is refined by the two-agent sliding window search. double The Euclidean distance D(x) between the two agents j(t) ,x i(t) Composed of ), i = 1, 2, and i ≠ j, R _j(t) It is specific to each agent and depends on the agent's actions and local observations; λ is a weighting factor used to adjust the trade-off between the overall and individual aspects of the reward value; I(P t ,L) is the penalty function.
[0068] Furthermore, the local refinement model predicts the label value L of the pixel at (x,y). x,y The segmentation probability P predicted by the local refinement model at location (x,y) x,y When the sum is 0, the penalty function gives a negative reward; otherwise, it gives a positive reward. The global penalty function I(P,L) for the segmentation result is:
[0069]
[0070] Where x, y represent the coordinates of a certain pixel when the window slides at time step t, and C represents the coordinates of the global pixels in the segmentation result.
[0071] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0072] This invention formulates image segmentation as a sequential decision-making problem based on multi-agent deep reinforcement learning: each pixel is considered as an agent cooperating with other agents, which share the same policy and utilize a dominant action commenting algorithm to update the policy. Based on this, a dual-agent structure is reconstructed to quickly obtain the coordinates of the segmentation target requiring refinement (i.e., the target to be segmented in a medical pathology image, hereinafter referred to as the image). A local refinement model is used to refine the local segmentation of the target to enhance segmentation performance. The agents can cooperate with each other through a graph convolutional network. The agents also provide rewards, defined as positive rewards if the agent's predicted action improves in a time step compared to the previous step, and negative rewards otherwise. Attached Figure Description
[0073] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0074] Figure 1 This is a flowchart of the method of the present invention;
[0075] Figure 2 This is a schematic diagram illustrating the operation of the intelligent agent according to an embodiment of the present invention;
[0076] Figure 3 This is a schematic diagram of the structure of the advantageous action commenting algorithm according to an embodiment of the present invention;
[0077] Figure 4 This is a schematic diagram illustrating the interaction between the intelligent agent and the environment in an embodiment of the present invention. Detailed Implementation
[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0079] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements or an indirect connection via other elements.
[0080] Example:
[0081] This invention formulates image segmentation as a sequential decision-making problem based on multi-agent deep reinforcement learning: each pixel is considered as an agent cooperating with other agents, which share the same policy and utilize a dominant action commenting algorithm to update the policy. Based on this, a dual-agent structure is reconstructed to quickly obtain the coordinates of the segmentation target requiring refinement (i.e., the target to be segmented in a medical pathology image, hereinafter referred to as the image). A local refinement model is used to refine the local segmentation of the target to enhance segmentation performance. The agents can cooperate with each other through a graph convolutional network. The agents also provide rewards, defined as positive rewards if the predicted action improves in a time step compared to the previous step, and negative rewards otherwise.
[0082] This invention is achieved through the following technical solution, please refer to [link / reference]. Figure 1 A pathological image segmentation method based on multi-agent deep reinforcement learning includes the following steps:
[0083] Step 1: Treat each pixel as an intelligent agent and collect the state dataset and similarity information of each pixel.
[0084] Treating each pixel in the image as an intelligent agent, for a medical pathology image containing N pixels x = (x1,...,x...) i ,...,x N ), i = 1, 2, ..., N, each pixel x i Treat it as a single intelligent agent and share policies with other intelligent agents. in, Indicates an action, Indicates the state.
[0085] Mapping states to corresponding actions, where states and actions can be defined as follows:
[0086] State (State Dataset): A concatenation of pixel values x from an image i The previous iteration step on pixel value x i Estimated probability The segmentation target predicted in the previous iteration step Define state Right now
[0087] Action: Action The set A = {A k}, k = 1, 2, ..., K, K contains K actions.
[0088] Pixel similarity information includes nearest neighbors in feature space, color space, and distance space. The feature space includes manually created features or features extracted by neural networks; the color space includes RGB and HSV colors; and the distance space includes Euclidean distance or geodesic distance. By combining this similarity information, the agent can understand the relationships between pixels in an image, improving the accuracy of image segmentation.
[0089] Step 2: Use deep reinforcement learning to estimate the probability that the pixel corresponding to the agent belongs to the segmentation target, and classify the pixel according to the estimated probability to determine whether the pixel belongs to the segmentation target.
[0090] Deep reinforcement learning is used to estimate the probability that a pixel belongs to the segmentation target for each agent, and the pixel is then classified based on the estimated probability to determine whether it belongs to the segmentation target. Each iteration of the deep reinforcement learning process is executed according to the agent's policy.
[0091] Please see Figure 2 The strategy is implemented using the Advantageous Action Commenting Algorithm Network (A2C). Make state and actions Perform mapping to obtain the pixel x i Estimated probability The agent adjusts the reward based on the probability estimated at each iteration step. When the probability estimated in this iteration step Probability compared to the previous iteration step When there is an improvement, a reward will be given. It is a positive value; when the probability estimated in this iteration step is... Probability compared to the previous iteration step When there is a decrease, the reward It is a negative value.
[0092] award It can be defined as the difference in cross-entropy loss:
[0093]
[0094]
[0095]
[0096] in, This represents the cross-entropy loss at iteration step t. The cross-entropy loss represents the loss at iteration step t+1; yi represents the loss of pixel x. i The truth value label.
[0097] For pixel x i After T iterations, the total reward is:
[0098]
[0099] Where, γ t-1 This represents the discount factor for iteration step t-1, ranging from 0 to 1.
[0100] Please see Figure 3 When using deep reinforcement learning to estimate the probability that a pixel belongs to a segmentation target, the state dataset of the pixel and the similarity based on that pixel are used as inputs to the Advantage Action Commenting (A2C) algorithm. The aim is to enable the agent to better predict the probability that a pixel belongs to a segmentation target by considering the states of neighboring pixels. Based on the state and similarity of neighboring pixels, a weight is assigned to each neighboring pixel, and then these weights are used to adjust the prediction results. Weighted adjustments are made to improve the accuracy of segmentation.
[0101] The Advantage Action Commenting (A2C) algorithm comprises a Graph Convolutional Network (GCN), a policy head, and a value head. The policy head includes a first convolutional block, a second convolutional block, a third convolutional block, a first upsampling layer, and a softmax layer, all connected in sequence. The value head includes a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, and a second upsampling layer, all connected in sequence. Each convolutional block may include a convolutional layer, a non-linear activation layer, and a batch normalization layer, all connected in sequence. Each upsampling layer may include a bilinear upsampling layer or a transposed convolutional layer, which expands the feature map size due to downsampling or pooling operations. The softmax layer outputs a vector representing the probability distribution of a list of potential outcomes.
[0102] In summary, graph convolutional networks extract information about the relationships between pixels based on similarity, while the policy head is used to predict the probability that a pixel belongs to the segmentation target. Based on the state of neighboring pixels and their similarity, a weight is assigned to each neighboring pixel, and these weights are then used to adjust the prediction results. After weighted adjustment, the value head estimates the current state. The value of.
[0103] Using θ g θ p θ v Let θ represent the model parameters of the graph convolutional network, policy head, and value head, respectively. During training, θ p θ v The gradient is calculated as follows:
[0104]
[0105]
[0106] in, Represents the differential operator; It is the probability that the i-th pixel at time step t, i.e., the i-th agent, chooses action a in state s; Represents the logarithmic probability value; γ k-t This represents the discount factor from time step t to time step k; V represents the reward obtained by pixel i at time step k; i (s (t) ) represents the state value function of pixel i.
[0107] Step 3: Repeat step 2 until all pixels in the image have undergone the above operation, and finally obtain the coarse segmentation result of the image.
[0108] Perform for each pixel Figure 2 , Figure 3 The process shown is as follows: after pixel 1 completes step 2, pixel 2 completes step 2, and so on, until all pixels in the image have completed step 2, resulting in a coarse segmentation result of the image. This segmentation result is rough and requires further refinement in step 4.
[0109] Step 4: Within the coarsely segmented target, a multi-agent deep reinforcement learning algorithm is used to re-determine two pixels as new agents. These agents quickly find the pixel coordinates that need to be refined based on the known coarse segmentation results, and use a local refinement model to process the pixels that need to be refined, ultimately obtaining a refined segmentation target.
[0110] Please see Figure 4This diagram illustrates the interaction between the agent and its environment. Two agents, Agent 1 and Agent 2, are constructed using a multi-agent deep reinforcement learning algorithm (MADDPG). These two agents monitor each other's states and actions, quickly finding the coordinates of the window to be refined (each window is located within one pixel) on the segmentation result known in step 3. For example, Agent 1 can slide row by row from left to right and top to bottom starting from the first pixel of the segmentation result, while Agent 2 can slide row by row from right to left and bottom to top starting from the last pixel of the segmentation result, quickly finding the coordinates of the window to be refined. If one agent finds the coordinates of the window to be refined, the other agent stops searching. The two agents share parameters in the convolutional layers to achieve knowledge sharing and have independent fully connected layers for decision-making. Using a dual-agent approach to quickly slide on the coarse segmentation result, they traverse the entire image, with each agent processing approximately half of the pixels. Compared to a single agent sliding, this method can quickly find the coordinates of the window to be refined.
[0111] The dual-agent system determines the pixel coordinates that need refinement based on the existing coarse segmentation results. These coordinates can be those of a single pixel or multiple pixels within a small region. The original medical pathology image is then concatenated with the coarse segmentation results along the channel dimension, serving as the observation state for the reinforcement learning model. The output of the dual-agent system represents the coordinates of the window in the environment that needs refinement.
[0112] As an example, the action space that each agent can take consists of a 12*5+1 dimensional vector. The first 12*5 dimensions of the vector represent the step size of the window the agent can move through, and the last dimension represents the sequence in which the agent chooses to stop. Therefore, the mathematical expressions for the action spaces of Agent 1 and Agent 2 are:
[0113] Action1 = {W 10 W 11 W 12 ,...,W 1(12*5-1) ,S1}
[0114] Action2 = {W 20 W 21 W 22 ,...,W 2(12*5-1) ,S2}
[0115] Where Action1 is the action space of agent one, S1 is the sequence for agent one to choose to stop, Action2 is the action space of agent two, S2 is the sequence for agent two to choose to stop, and W... ji Let i represent the different step sizes of the window that agent j can move, where j = 1, 2, i = 0, 1, ..., (12 * 5 - 1).
[0116] N y This represents the number of sliding windows the agent makes along the y-axis of the image. The window number i and its corresponding coordinates are:
[0117]
[0118] y = (i-1) mod N y
[0119] Where x and y represent the horizontal and vertical coordinates of the window.
[0120] The policy function π(a|s) is used to define the policy adopted by the agent. The mathematical expression of the policy function is as follows:
[0121] π(a|s)=P(A t =a|S t =s)
[0122] When agent 1 takes action a1 in state s, the expected reward is:
[0123]
[0124] The expected reward r1(s,a1) represents the immediate reward of agent 1 in state s given agent 2's policy π2(a2|s) and state transition probability p(s`|s,a1,a2). In this way, agent 1 can consider the impact of agent 2's policy and state transition probability on its own policy and behavior, thereby achieving better cooperation and collaboration.
[0125] Given all possible actions a2 of agent 2, the probability that the environment transitions from state s to state s' after agent 1 takes action a1 is:
[0126]
[0127] The above formula calculates the probability of state transition given agent 1's action a1 and agent 2's policy π2(a2|s) by weighted summation over all possible actions of agent 2. In this way, agent 1 considers the influence of agent 2's policy on the state transition probability, thus more accurately predicting state transitions and making corresponding decisions.
[0128] The gradient of the expected return of agent j is:
[0129]
[0130] Wherein, J(θ) j The policy performance of agent j is represented by the expected reward, which is related to the policy parameters θ of agent j. j The function; logπj (a j |s j ) indicates that in state s j Below, agent j selects action a. j The logarithmic probability; This represents the policy π of a given agent j. j Given other agent policies, the state x and the values of actions a1 and a2 chosen by Agent 1 and Agent 2 are function functions; where state x = (o1, o2), o N (N∈[1,2]) represents the observations of agent j (j=1,2) at the current time step. The observations contain all the information acquired by agent j in the current state. This information includes, but is not limited to: the current state, the states of other agents, the actions of other agents, rewards, etc. The agent makes decisions and selects actions based on the observations in order to optimize its own strategy.
[0131] The two agents can select the coordinates of a window in the environment through their own actions and information interaction with each other. The environment then uses a local refinement model to optimize the region selected by the agent and provides a corresponding reward value. This process continues to iterate until the agent outputs a stop signal.
[0132] The environment in which the two agents exist can be viewed as a tuple as shown below:
[0133] Env = {Image, L Global ,P Global Pos x,y ,P Local ,L Local}
[0134] Where Image is the original medical pathology image, and L Global P is the original label given by the experts. Global Pos is the probability of segmenting the current image. x,y P selects the coordinates of the region for the agent. Local L is the segmentation probability for selecting a region for the agent. Local The label results for selecting regions for the agent. Among them, Image and P... Global The state is visible to the agent, and the label data L (L includes L) Global and L Local Pos is a hidden state that is not visible to the agent. x,y P Local This is the output of the intelligent agent.
[0135] During the interaction between the agent and the environment, the reward value fed back by the environment is R. t :
[0136]
[0137] R _j(t) =P double +D(x j(t) ,x i(t) )
[0138] Among them, R t The reward value is the value obtained at time step t; the Dice coefficient is used to measure the segmentation probability image P at time step t. t Similarity between the data and the label data L; R _j(t) Let P be the individual reward obtained by agent j at time step t, where j = 1, 2, and P be the segmentation target estimation probability P of the pixel when the pixel is refined by the two-agent sliding window search. double The Euclidean distance D(x) between the two agents j(t) ,x i(t) Composed of ), i = 1, 2, and i ≠ j, R _j(t) It is specific to each agent and depends on the agent's actions and local observations; λ is a weighting factor used to adjust the trade-off between the overall and individual aspects of the reward value; I(P t ,L) is the penalty function.
[0139] The predicted label value L of the pixel at (x,y) using the local thinning model x,y The segmentation probability P predicted by the local refinement model at location (x,y) x,y When the sum is 0, the penalty function gives a negative reward; otherwise, it gives a positive reward. The global penalty function I(P,L) for the segmentation result is:
[0140]
[0141] Where x, y represent the coordinates of a certain pixel when the window slides at time step t, and C represents the coordinates of the global pixels in the segmentation result.
[0142] When the reward value from the environment feedback is positive, the region replacement module can use this positive reward value to inform Agent 1 and Agent 2 to continue interacting and find the next region that needs to be refined. A region can consist of one or more pixels. After the agent finds the region that needs to be refined, it outputs the coordinates of the window to the environment until there are no more pixels that need to be refined in the known coarse segmentation results. This is the final segmentation target, which is a refined segmentation target.
[0143] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A pathological image segmentation method based on multi-agent deep reinforcement learning, characterized in that: Includes the following steps: Step 1: Treat each pixel as an intelligent agent and collect the state dataset and similarity information of each pixel; Step 2: Use deep reinforcement learning to estimate the probability that the pixel corresponding to the agent belongs to the segmentation target, and classify the pixel according to the estimated probability to determine whether the pixel belongs to the segmentation target. When using deep reinforcement learning to estimate the probability that a pixel belongs to a segmentation target, the state dataset of the pixel and the similarity based on the pixel are used as inputs to the dominant action commenting algorithm. This allows the agent to better predict the probability that a pixel belongs to a segmentation target by using the states of neighboring pixels. Based on the states and similarities of neighboring pixels, a weight is assigned to each neighboring pixel, and these weights are used to adjust the prediction results, thereby improving the accuracy of segmentation. Step 3: Repeat step 2 until all pixels in the image have undergone the above operation, and finally obtain the coarse segmentation result of the image; Step 4: Within the coarsely segmented target, a multi-agent deep reinforcement learning algorithm is used to re-determine two pixels as new agents. These agents quickly find the pixel coordinates that need to be refined based on the known coarse segmentation results, and use a local refinement model to process the pixels that need to be refined, ultimately obtaining a refined segmentation target.
2. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In step 1, the step of treating each pixel as an agent includes: For a medical pathology image containing N pixels Let i = 1, 2, ..., N, and each pixel x i Treat it as a single intelligent agent and share policies with other intelligent agents. ,in, Indicates an action, Indicates state; By concatenating the pixel values x of the image i The previous iteration step on pixel value x i Estimated probability The segmentation target predicted in the previous iteration step Define state ; action The set A = {A k }, k=1,2,...,K, K contains K actions.
3. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In step 1, the pixel similarity information includes the nearest neighbor points in the feature space, color space, and distance space. The feature space includes manually created features or features extracted by a neural network; the color space includes RGB space and HSV space; and the distance space includes Euclidean distance or geodesic distance.
4. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 2, characterized in that: Step 2, which involves estimating the agent using deep reinforcement learning and predicting the probability that the pixel corresponding to the agent belongs to the segmentation target, includes: Execution strategy using the superior action commenting algorithm network. , to make the state and actions Perform mapping to obtain the pixel x i Estimated probability ; The agent adjusts the reward based on the probability estimated at each iteration step. When the probability estimated in this iteration step Probability compared to the previous iteration step When there is an improvement, a reward will be given. It is a positive value; when the probability estimated in this iteration step is... Probability compared to the previous iteration step When there is a decrease, the reward It is a negative value.
5. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 4, characterized in that: The reward The difference in cross-entropy loss: in, This represents the cross-entropy loss at iteration step t. y represents the cross-entropy loss at iteration step t+1; i Represents pixel x i The truth value label.
6. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 5, characterized in that: For pixel x i After T iterations, the total reward is: in, This represents the discount factor for iteration step t-1, ranging from 0 to 1.
7. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 6, characterized in that: In step 2, the dominant action commenting algorithm includes a graph convolutional network, a policy head, and a value head; the graph convolutional network extracts information about the relationships between pixels based on pixel similarity; the policy head is used to predict the probability that a pixel belongs to the segmentation target. Based on the state and similarity of neighboring pixels, a weight is assigned to each neighboring pixel, and then these weights are used to adjust the prediction results. A weighted adjustment is performed; the value head is used to estimate the current state. Value; The strategy head includes a first convolutional block, a second convolutional block, a third convolutional block, a first upsampling layer, and a softmax layer connected in sequence; the value head includes a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, and a second upsampling layer connected in sequence. Each convolutional block consists of a convolutional layer, a non-linear activation layer, and a batch normalization layer connected in sequence; Each upsampling layer consists of either a bilinear upsampling layer or a transposed convolutional layer.
8. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 7, characterized in that: use Let represent the model parameters of the graph convolutional network, policy head, and value head, respectively, during training. The gradient is calculated as follows: in, Represents the differential operator; It is the probability that the i-th pixel at time step t, i.e., the i-th agent, chooses action a in state s; Represents the logarithmic probability value; This represents the discount factor from time step t to time step k; This represents the reward that pixel i receives at time step k; The function representing the state value of pixel i.
9. The pathological image segmentation method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In step 4, within the coarsely segmented target, a multi-agent deep reinforcement learning algorithm is used to re-determine two pixels as new agents. These agents quickly search for the pixel coordinates that need to be refined based on the known coarse segmentation results. This includes: Two agents, Agent 1 and Agent 2, are constructed using a multi-agent deep reinforcement learning algorithm. The two agents pay attention to each other's state and actions. The window coordinates that need to be refined are found on the coarse segmentation results known in step 3. The action space available to each agent consists of an M+1 dimensional vector. The first M dimensions of the vector represent the step size of the window the agent can move through, and the last dimension represents the sequence in which the agent chooses to stop. Therefore, the mathematical expressions for the action spaces of Agent 1 and Agent 2 are: Where Action1 is the action space of agent one, S1 is the sequence for agent one to choose to stop, Action2 is the action space of agent two, S2 is the sequence for agent two to choose to stop, and W... ji Let i represent the different step sizes of the window that agent j can move, where j=1,2, i=0,1,...,(M-1); N y This represents the number of sliding windows the agent makes along the y-axis of the image. The window number i and its corresponding coordinates are: Where x and y represent the horizontal and vertical coordinates of the window.