A multi-agent and environment interaction learning method for image key point detection
By employing a multi-agent interaction learning method with the environment, combined with Markov decision processes and multi-scale search strategies, the accuracy problem of image keypoint detection under illumination changes and target occlusion was solved, achieving efficient and accurate keypoint localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2026-03-26
- Publication Date
- 2026-06-05
Smart Images
Figure CN122156595A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image detection technology, and in particular to a multi-agent and environment interaction learning method for image key point detection. Background Technology
[0002] Image key point detection is one of the core tasks in the field of image analysis.
[0003] In recent years, the automation level of this task has been significantly improved, but existing methods still have several shortcomings that urgently need to be addressed. First, existing solutions use predefined fixed search patterns such as sliding windows and full-image dense prediction when locating key points through coordinate regression and heatmap prediction. However, the model training in existing methods relies on a large amount of accurately labeled key point datasets, but the detection accuracy of the trained models is difficult to meet practical needs due to drastic changes in lighting, partial occlusion of targets, and diversity of poses. Summary of the Invention
[0004] Therefore, it is necessary to provide a multi-agent and environment interaction learning method for image key point detection to address the above-mentioned technical problems. This method utilizes an environment interaction feedback mechanism to improve the detection accuracy of image key points under conditions of diverse scenarios and limited computing resources.
[0005] The following technical solution is adopted in this specification: This specification provides a multi-agent environment interaction learning method for image keypoint detection. The target image serves as the environment for the agents, and each agent represents a movable point on the target image. Multiple agents are present on the target image. The method includes: Obtain the current state of each agent; the state of an agent includes the pixel values of its field of view in the target image at multiple consecutive time points; For any agent, the action of the agent is decided by the behavior network based on the agent's current state, and the action to be taken at the current moment is determined. The position of the agent at the next moment is determined based on the action and the step size. The behavior network is the agent action decision module in the Markov decision process, which is responsible for the interaction learning between the agent and the environment. Its network parameters are iteratively optimized through reinforcement learning training. If the agent has the same position at multiple moments within a consecutive preset time period, the agent's current scale search level is adjusted to the next scale search level. The agent's state at the next moment is determined based on the agent's position at the next moment and the adjusted field of view. Each scale search level includes the field of view and the movement step size. The field of view and movement step size of the previous scale search level are both greater than those of the next scale search level. If the agent's position is not the same at multiple moments within a consecutive preset time period, the state of the agent at the next moment is determined based on the agent's position at the next moment and the current field of vision. Based on the agent's state at the next moment, the action of the agent is decided by the behavior network until the agent's position is the same at multiple moments within a consecutive preset time period under the minimum scale search level. The final positions of all agents are determined as key points on the target image.
[0006] Optionally, each agent corresponds to a behavior network, which includes convolutional layers, pooling layers, and fully connected layers. All agents share the same set of convolutional and pooling layer network parameters for extracting general features of the image. Each agent has its own independent set of fully connected layer parameters, and there is an interactive communication mechanism between the fully connected layers of different agents. The specific implementation process of the interactive communication mechanism is as follows: For any adjacent upper and lower fully connected layers of the agent, obtain the output features of the upper fully connected layer of the current agent, and at the same time obtain the output features of the upper fully connected layers of other agents at the same level, and calculate the mean feature vector of the output features of all upper fully connected layers; concatenate the mean feature vector with the output features of the upper fully connected layer of the current agent along the channel dimension; use the concatenated feature vector as the input feature of the lower fully connected layer of the current agent.
[0007] Optionally, the Markov decision process includes a set of states, a set of actions, a reward function, and a reward discount factor; The state set includes: selecting the pixel values of the agent's field of vision at multiple consecutive time points, stitching them together along the channel dimension, and using the resulting feature map as the agent's current state; The action set includes: designing the agent's actions as discrete two-dimensional movement actions, including along... Move along the positive axis Move along the negative axis Move along the positive axis Move in the negative direction of the axis; The reward function includes: ; in, The reward value for the agent. Perform actions for the agent Distance between the front and the target key point Perform actions for the agent The distance from the target key point For reward scaling factor; Reward discount factors include: , Used to balance the weight of current rewards and future rewards in cumulative rewards: when When =0, the cumulative reward is related to the current reward; when When the value is 1, the current reward and the future reward have equal weight in the cumulative reward.
[0008] Optionally, the training process of the behavior network includes: The initial behavior network is trained by drawing multiple sets of experience samples from the experience replay buffer, and its parameters are iteratively updated. The initial behavior network after training is defined as the behavior network. During training, a multi-scale search strategy is employed to alter the field of vision and step size of the sample agents, changing them from a large field of vision and large step size to a small field of vision and small step size; at each interval... N The next step is to copy the updated parameters of the initial behavioral network to the target network to update the parameters of the target network. The target network and the behavior network have the same structure, but their parameter update logic is independent of each other.
[0009] Optionally, the loss function used during the training of the behavioral network. for: ; in, As a reward discount factor, Selecting the first for the behavioral network Moment State action , Calculate actions for the target network value, Calculate the first behavior network. Moment Action value, To reward scaling factor, Perform actions for the agent Distance between the front and the target key point Perform actions for the agent The distance between the target key point and the target.
[0010] Optionally, before training the initial behavior network, the method further includes: The initial sample state is obtained by acquiring the pixel values of the sample agent's field of view at multiple consecutive time points in the sample image; there are multiple sample agents. Based on the initial sample state, the initial behavior network is used to make decisions on the actions of the sample agent and determine the actions of the sample agent. Execute the action and movement step of the sample agent to obtain the position of the sample agent at the next moment; Using Markov decision processes, the reward value is calculated based on the initial position and next time step position of the sample agent, as well as the position of the corresponding key point of the sample. The next sample state is obtained and the quadruple is placed in the experience replay buffer. The quadruple includes the initial sample state, action, reward value and next sample state of the sample agent at the current time step. Based on the next sample state of the sample agent, continue the search to obtain a new sample state and generate a new quadruple, which is then stored in the experience replay cache.
[0011] Optionally, the method further includes: During the parameter update of the initial behavior network, the updated initial behavior network parameters are used simultaneously to recalculate the predicted actions corresponding to the sample states in the experience replay buffer, construct an updated quadruple containing the predicted actions, and replace the corresponding original quadruple in the experience replay buffer with the updated quadruple.
[0012] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: The agent action decision module in the Markov decision process of the behavior network provided by this invention is responsible for the interaction between the agent and the environment. Its network parameters are iteratively optimized through reinforcement learning training. The action of the agent is determined through the behavior network, which improves the detection accuracy. Furthermore, in the keypoint detection process, a multi-scale search strategy is introduced. The switching condition is that the agent's position is the same at multiple moments within a consecutive preset time period, adjusting the large-scale search level to a small-scale search level, thus balancing detection efficiency and localization accuracy. This method can more fully capture the keypoint features in the target image by combining multi-agent interaction with the environment and a multi-scale search strategy. In scenarios with diverse scenes and limited computing resources, the accuracy of keypoint localization is significantly improved compared to existing methods.
[0013] Furthermore, this invention proposes a Markov decision process design adapted for image key point detection, which uses the pixel values of the agent's field of view at multiple consecutive moments on the target image as the state and the Euclidean distance change as the reward function to achieve accurate interaction between the agent and the image environment.
[0014] Furthermore, all agents share the same set of convolutional and pooling layer network parameters, enabling different agents to perceive logically consistent image features, avoiding decision conflicts caused by differences in feature extraction logic, while reducing the number of parameters and improving training efficiency. An interactive communication mechanism is established between the fully connected layers of the agents. Specifically, for any adjacent upper and lower fully connected layers in the behavior network, the mean feature vector of the output features of all upper fully connected layers is calculated, and this mean feature vector is concatenated to the output features of each agent's upper fully connected layer, serving as the input features of the agent's lower fully connected layer. In this way, each agent can utilize the decision information of other agents, allowing each agent to integrate decision information from different perspectives, achieving a relative spatial distribution between key points, thereby improving detection accuracy.
[0015] In addition, the loss function used throughout the training process of the behavior network separates action selection and value calculation, accurately measuring the difference between the predicted value output by the current behavior network and the target value calculated by the target network, thereby improving the accuracy of the behavior network in predicting actions. Attached Figure Description
[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0017] Figure 1 This document provides a flowchart of a multi-agent interaction learning method for image keypoint detection. Figure 2 This is a schematic diagram of the architecture of a behavior network corresponding to a multi-agent system provided in this specification. Figure 3 This document provides a schematic diagram of a multi-agent and environment interaction learning method for image keypoint detection. Figure 4 A schematic diagram of a Markov decision process provided in this specification; Figure 5 This is a schematic diagram of an experimental effect provided in this instruction manual; Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.
[0019] This application provides a multi-agent, environment-interaction learning method for image keypoint detection. This method differs from traditional methods such as full-image regression by employing an agent-environment interaction strategy. The agent learns the optimal exploration path to effectively locate keypoints in the image, and the reward value serves as guidance, continuously optimizing and moving closer to the keypoints. This invention uses multi-agent reinforcement learning to model the detection task, generating effective training samples without requiring large-scale labeled data, a unique technical approach. Simultaneously, it avoids redundant computation, reduces dependence on computational resources and labeled data, achieves accurate detection under limited conditions, and exhibits excellent robustness and generalization ability.
[0020] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0021] Figure 1 This is a flowchart of a multi-agent interaction learning method for image keypoint detection, as described in this specification. The target image represents the environment of the agents, and each agent represents a movable point on the target image. The target image includes multiple agents. The method specifically includes the following steps: S101, obtain the current state of each agent; the state of an agent includes the pixel values of its field of view in the target image at multiple consecutive time points.
[0022] For any agent, at the initial moment of keypoint detection, the agent's positions on the target image can be randomly selected across multiple consecutive time points. The pixel values of the agent's field of view at the corresponding positions on the target image at these consecutive time points are then concatenated along the channel dimension. The concatenated result is used as the agent's state at the corresponding time point. The field of view can be rectangular.
[0023] Optionally, the pixel values of the agent's field of vision from four consecutive moments between the historical moment and the current moment can be stitched together in the channel dimension, and the stitching result can be used as the agent's current state.
[0024] S102, for any agent, based on the agent's current state, the action of the agent is decided through the behavior network, determining the action to be taken at the current moment, and the position of the agent at the next moment is determined based on the action and the step size. The behavior network is the agent action decision module in the Markov decision process, which is responsible for the interaction learning between the agent and the environment. Its network parameters are iteratively optimized through reinforcement learning training.
[0025] The agent's current state is input into the behavior network, which outputs the expected cumulative reward value (i.e., value) for each action in the action set. The action with the highest value is determined as the action taken by the agent at the current moment. Then, the agent is controlled to move in the corresponding direction by the corresponding movement step size from the current position to obtain the agent's position at the next moment.
[0026] In one embodiment, for a target image keypoint detection task, the present invention constructs a Markov decision process, which includes a set of states. Action set Reward function and reward discount factor The specific design is as follows: State set This includes: selecting pixel values corresponding to the agent's field of vision at multiple consecutive time points, stitching them together along the channel dimension, and using the stitched feature map as the agent's current state.
[0027] Specifically, the state of the agent is defined as "the pixel value corresponding to the rectangular field of view centered on the agent's own position". To ensure the continuity of the state, a field of view buffer is set up to concatenate the pixel values of the agent's field of view corresponding to four consecutive time steps in the channel dimension. The concatenated result is used as the state of the agent.
[0028] Action set This includes: designing the agent's actions as discrete two-dimensional movement actions, including along... Move in the positive direction of the axis ( ),along Movement in the negative direction of the axis ( ),along Move in the positive direction of the axis ( ),along Movement in the negative direction of the axis ( Action selection strategy adopts The strategy, expressed as:
[0029] (1); in, This means randomly selecting an action from the action set. This represents the probability of choosing the corresponding action. This represents the probability of randomly selecting an action. Indicates in The action chosen at any moment Indicates time The corresponding state, express Constant through behavioral networks Choose the most valuable movements. During the training phase, The value decreases linearly to 0.1 with each training round to balance the agent's environment exploration and strategy utilization; in the application phase, A value of 0 indicates that the agent uses a greedy strategy to select the optimal action.
[0030] Action set Provide the agent with a basic and complete set of motion instructions, enabling it to actively explore and navigate to target key points in the image environment through sequential movement.
[0031] The reward function is defined as the difference in distance between the agent and the target keypoint before and after performing an action, multiplied by a reward scaling factor. The reward function includes: (2); in, The reward value for the agent. Perform actions for the agent Distance between the front and the target key point Perform actions for the agent The distance from the target key point The reward scaling factor; when the agent approaches the target key point, Take a positive value; when the agent moves away from the target key point, Take the negative value.
[0032] In this process, the calculation is performed first. Distance between the agent and the target key point at any given time : (3); in, for The coordinates of the agent at any given moment. The coordinates of the target key points.
[0033] The agent continuously updates its action network for finding key points based on feedback from reward values in order to achieve the task objective.
[0034] Reward discount factors include: , Used to balance the weight of current rewards and future rewards in cumulative rewards: when When =0, the cumulative reward is related to the current reward; when When the value is 1, the current reward and the future reward have equal weight in the cumulative reward.
[0035] In one embodiment, the dual-depth network architecture includes a behavior network and a target network; the behavior network and the target network have the same structure, but their parameter update logic is independent of each other; the behavior network includes convolutional layers, pooling layers, and fully connected layers.
[0036] Optionally, the behavior network can be designed based on ResNet18 as a network architecture with shared convolutional layers, pooling layers, independent fully connected layers, and multi-agent collaboration. Through hierarchical functional division and agent collaboration mechanism, it can achieve coordinated optimization of feature extraction, decision output, and global information fusion.
[0037] Each agent corresponds to a behavior network Q and a target network Q'. The two networks have the same structure but separate parameter update logic. The core of the solution is to address the problem of "target value fluctuation" in reinforcement learning.
[0038] (1) Feature extraction module: It is built based on the ResNet18 network and contains multiple sets of convolutional layers and pooling layers. The purpose is to extract low-level features of the image from the input agent's field of vision. ResNet18 is chosen as the basis because its residual connection structure can effectively alleviate the gradient vanishing problem of deep networks, ensure the full extraction of low-level features, and provide reliable feature support for subsequent action decisions.
[0039] (2) Fully connected layer: The high-dimensional features output by the convolutional layer are mapped to the Q-values (i.e., the cumulative expected reward value of each action) corresponding to the four discrete actions. The purpose is to transform the low-level features into the decision basis of the agent. The independent design of the fully connected layer realizes the mapping from "features" to "action value", ensuring that each agent can output personalized action decisions according to its own field of vision, and adapt to the search needs of different key points.
[0040] The dual-depth network architecture design not only ensures the effective extraction of image features and realizes personalized output of action decisions, but also lays the foundation for subsequent multi-agent collaboration and behavioral network training.
[0041] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the behavior network architecture corresponding to multiple agents. All agents share the same set of convolutional and pooling layer network parameters to extract general features of the image. Multiple convolutional and pooling layers constitute the feature extraction module. The implementation process of the feature extraction module includes: first, using a 7x7 convolutional layer (Conv, 7x7, 64) to output 64 channels, and then passing it through a 3x3 max pooling layer (MaxPool3x3); there are 4 sets of convolutional layers in the middle, each set of convolutional layers includes 2 layers of 3x3 convolutional layers, and the output channels of the 4 sets of convolutional layers are 64, 128, 256 and 512 respectively; finally, it is output through a 1x1 average pooling (AvgPool 1x1); the behavior network includes 4 fully connected layers (FC).
[0042] Each agent possesses an independent set of fully connected layer parameters, and there is an interactive communication mechanism between the fully connected layers of different agents. The specific implementation of this interactive communication mechanism involves obtaining the output features of the current agent's previous fully connected layer for any adjacent upper and lower fully connected layers, simultaneously obtaining the output features of other agents' previous fully connected layers at the same level, and calculating the mean feature vector of all the output features of the previous fully connected layers. This mean feature vector is then concatenated with the output features of the current agent's previous fully connected layer along the channel dimension. The concatenated feature vector is then used as the input feature of the current agent's next fully connected layer.
[0043] Specifically, this embodiment designs a multi-agent collaboration mechanism: Since the field of vision of a single agent is limited, it is difficult to utilize the global features of the image (such as the spatial distribution between key points and the overall anatomical structure). This embodiment designs a multi-agent collaboration mechanism of "implicit communication + explicit communication" to allow multiple agents to share information and complement each other's fields of vision.
[0044] (1) Implicit communication: All agents share the same set of convolutional and pooling layer parameters in their behavioral networks. The purpose is to ensure that each agent has a consistent logic for extracting low-level features of the image. For example, if the target image is a medical image, the anatomical structure of the medical image (such as bone morphology and organ contours) has fixed spatial features. Sharing convolutional layers can enable different agents to perceive consistent anatomical structure information, avoid decision conflicts caused by differences in feature extraction logic, and at the same time reduce the number of parameters and improve training efficiency.
[0045] (2) Explicit Communication: The average output of the fully connected layer is concatenated. After each agent's fully connected layer output is calculated independently, the average of the fully connected layer outputs of all agents is taken, and then the average is concatenated into the input of the next fully connected layer. The purpose is to allow agents to utilize the decision information of other agents. Fusion of global features: The fully connected layer output of a single agent only reflects local information within its own field of vision, while the average output of multiple agents integrates decision information from different fields of vision, reflecting the relative spatial distribution between key points.
[0046] This collaborative mechanism employs shared convolutional and pooling layers for feature extraction, then transforms the extracted features into fully connected layer mean concatenation to fuse global decisions. This process achieves information integration from local viewpoints to global features, solving the problem of limited viewpoints for a single agent, while improving the spatial distribution accuracy of keypoint detection.
[0047] In traditional network algorithms, the behavior network is responsible for both "selecting the optimal action" and "calculating the action value," which can easily lead to overestimation of action value (i.e., Q-value higher than the actual value), affecting training stability and localization accuracy. Therefore, to measure the difference between the predicted Q-value output by the current behavior network and the target Q-value calculated by the target network, an improved loss function is used throughout the network training process to accurately evaluate the value of each action taken in a specific state, thereby outputting the optimal action strategy for keypoint detection. Specifically, this application employs a dual-depth network and designs an improved loss function that separates "action selection" from "value calculation." Specifically, the loss function used by the behavior network during training... for:
[0048] (4); in, As a reward discount factor, Select the first for the behavioral network Q Moment State action The behavioral network Q interacts with the environment in real time and can capture the latest state-action relationships; Calculate actions for the target network Q' value, For behavioral network computation Moment Action Value; For the parameters of the behavioral network Q, The parameters are for the target network Q'.
[0049] In one embodiment, prior to training the initial behavior network, the embodiment includes the following steps: S201, obtain the pixel values of the sample agent's field of view at multiple consecutive time points on the sample image as the initial sample state; there are multiple sample agents.
[0050] S202, Based on the initial sample state, the initial behavior network is used to make decisions on the actions of the sample agent and determine the actions of the sample agent.
[0051] S203, execute the action and movement step of the sample agent to obtain the next position of the sample agent.
[0052] S204. Using a Markov decision process, the reward value is calculated based on the initial position and next position of the sample agent, as well as the position of the corresponding key point of the sample, and the next sample state is obtained. The quadruple is placed in the experience replay buffer. The quadruple includes the initial sample state, action, reward value and next sample state of the sample agent at the current moment.
[0053] S205, based on the next sample state of the sample agent, continue the search to obtain a new sample state and generate a new quadruple to store in the experience replay buffer.
[0054] In one embodiment, the training process of the behavior network includes: training an initial behavior network using multiple sets of experience samples drawn from an experience replay buffer to iteratively update the parameters of the initial behavior network. The initial behavior network after training is defined as the behavior network; the target network and the behavior network have the same structure and their parameter update logic is independent of each other; the experience samples are quadruplets stored in the experience replay buffer.
[0055] During training, a multi-scale search strategy is employed to alter the field of vision and step size of the sample agent, changing it from a large field of vision and large step size to a small field of vision and small step size. Every N steps, the updated parameters of the initial behavioral network are copied to the target network to update the parameters of the target network. .
[0056] In one embodiment, during the parameter update of the initial behavior network, the updated initial behavior network parameters can be used simultaneously to recalculate the predicted action corresponding to the sample state in the experience replay buffer, construct an updated quadruple containing the predicted action, and replace the corresponding original quadruple in the experience replay buffer with the updated quadruple.
[0057] In this embodiment, the separation logic of "behavioral network Q selects actions + target network Q' calculates value" avoids overfitting in traditional networks. Simultaneously, the loss function incorporates "distance change before and after the action (…)". , ) and "future reward discount" This ensures that the agent's decision-making considers both "current distance optimization" and "long-term reward accumulation".
[0058] In summary, the key point detection method provided in this embodiment, through the design concept of multi-agent interaction and learning with the environment, not only ensures the personalized decision-making of a single agent, but also realizes global information sharing among multiple agents. At the same time, it solves the stability problem in reinforcement learning training, providing network architecture support for efficient and accurate image key point detection.
[0059] S103, if the agent has the same position at multiple moments within a consecutive preset time period, the agent's current scale search level is adjusted to the next scale search level. The agent's state at the next moment is determined based on the agent's position at the next moment and the adjusted field of view. Each scale search level includes the field of view and the movement step size. The field of view and movement step size of the previous scale search level are both greater than the field of view and movement step size of the next scale search level.
[0060] In one embodiment, a total of [number] within a consecutive preset time period N At that moment, if N Within a certain time period, there are M If the agent's position is the same at all times, it means that "the agent's position is the same at multiple times within a consecutive preset time period," and the agent's current scale search level is adjusted to the next scale search level; if N Any time within a given moment M If at any given moment, the agent is not in the same position for all of the time points, then it means that "at multiple moments within a consecutive preset time period, the agent is not in the same position for all of the time points." M Less than N .
[0061] For example, with N It is 6. M Taking 3 as an example, at the first moment, the agent is at position 1. At the second moment, the agent moves to position 2. At the third moment, the agent moves back to position 1. At the fourth moment, the agent moves to position 3. At the fifth moment, the agent moves to position 4. At the sixth moment, the agent moves back to position 1. That is, within 6 consecutive moments, the agent is at position 1 for 3 moments. The agent satisfies the condition that "the agent has the same position at multiple moments within a consecutive preset time period".
[0062] The sequential decision-making properties of Markov models support the proposed multi-scale search strategy. The scale search levels, from top to bottom, include large field of view and large step size, medium field of view and medium step size, and small field of view and small step size. The agent uses a large field of view and large step size for "coarse localization" in early steps, and switches to a small field of view and small step size for "fine localization" in later steps, achieving a coarse-to-fine keypoint detection. This includes setting scale parameters and implementing scale switching logic.
[0063] Scale parameter settings: Based on the global structure and local texture feature requirements of the image, three sets of matching view scales and motion step sizes are set: (1) Field of view scale: Define three types of field of view (large field of view > medium field of view > small field of view), of which the large field of view (can cover more image areas) and the small field of view can focus on the details around key points.
[0064] (2) Movement step size: Three movement step sizes are set: large movement step size 3mm, medium movement step size 2mm, and small movement step size 1mm. Large step size can quickly traverse the image space, while small step size can achieve fine position adjustment.
[0065] This step allows the agent to match the "exploration range - movement efficiency" requirements at different detection stages, avoiding the problems of "slow global exploration" or "coarse local localization" under a single scale.
[0066] Scale switching logic: To accurately detect the location of key points, a phased scale switching process is designed, with the connection between each search phase based on the agent's search state: (1) Initial search phase: A large field of view and a large step size (3mm) are used. In this phase, the agent has no prior information about the location of key points. The large field of view can capture the global anatomical structure of the image, and the large step size can quickly narrow down the candidate area of key points, thus completing coarse localization. This enables the approximate range of key points to be locked in a short time, avoiding ineffective searches in local areas.
[0067] (2) Intermediate search phase: switch to medium field of view and medium step size (2mm). When the agent has approached the general area of the key point, narrowing the field of view can reduce interference from irrelevant areas, and reducing the step size can refine the search range, realizing the transition from "global exploration" to "local focus".
[0068] (3) Final search stage: A small field of view and a small step size (1mm) are used. At this time, the agent is close to the key point. The small field of view can focus on the local texture features of the key point, and the small step size can achieve pixel-level position adjustment to complete the fine localization.
[0069] (4) Switching conditions: A historical state buffer is set to record the position sequence of the agent. If the agent's position is the same at multiple times within a consecutive preset time period (determined as "oscillation"), it means that the position cannot be further optimized at the current scale search level. At this time, the current scale search level is ended and switched to a smaller scale search level. When oscillation occurs at the smallest scale search level (small field of view and small movement step size), it indicates that accurate localization has been completed and the detection process is terminated. For example, appearing at the same position 3 times within a consecutive preset time period can indicate that oscillation has occurred.
[0070] This strategy balances the "exploration efficiency" and "positioning accuracy" at different detection stages by using a progressive approach of "expanding the field of view to narrowing the field of view" and "increasing the step size to decreasing the step size," ultimately achieving efficient and accurate key point detection.
[0071] After adjusting the agent's current scale search level to the next scale search level, obtain the pixel values corresponding to the adjusted field of view centered on the agent's position at the next moment, thereby determining the agent's state at the next moment.
[0072] S104. If, within a consecutive preset time period, the agent does not have the same position at multiple moments, the state of the agent at the next moment is determined based on the agent's position at the next moment and its current field of vision.
[0073] If the agent's position is not the same at any of the multiple moments within a consecutive preset time period, the state of the agent at the next moment is determined based on the agent's position at the next moment and the field of vision of the current scale search level.
[0074] S105, based on the agent's state at the next moment, continue to make decisions on the agent's actions through the behavior network until the agent's position is the same at multiple moments within a consecutive preset time period under the minimum scale search level; determine the final position of all agents as key points on the target image.
[0075] Based on the state of the agent at the next moment, continue executing S102-S104 until the position of the agent remains unchanged for multiple consecutive moments at the minimum scale search level, and finally determine the final position of all agents as key points on the target image.
[0076] In one embodiment, the present invention also provides a multi-agent interaction learning method for image keypoint detection. This method uses the entire image as the interaction environment for the agents, defines the agents as movable points, and achieves keypoint detection through a dual-deep network algorithm of multi-agent reinforcement learning. Figure 3 As shown. The core process is as follows: 1) Agent-environment interaction process: Construct a Markov decision process quadruple containing state, action, reward function, and discount factor. The agent's initial position is random, and the state is the pixel value of the field of view at 4 consecutive time steps. The ε-greed strategy selects actions from four discrete actions: up, down, left, and right. It calculates the Euclidean distance between the agent and the key points and outputs the reward value. Synchronously update the next state Complete the interaction and use the quadruple obtained from multiple rounds of interaction. 1) Stored in the experience replay cache; 2) Multi-scale search strategy: The agent configures the field of view and step size in stages. Initially, it uses a large field of view and a large step size for coarse localization of key points. Later, it switches to a small field of view and a small step size for fine localization. The scale switching condition is "three times in the same position in a short period of time". The detection is terminated when the smallest scale oscillation occurs; 3) Dual-deep network training: A behavioral network is used. With the target network The dual architecture trains the behavior network by extracting experience samples from the experience replay buffer and updates the network parameters via gradient descent, while the target network... The behavior network parameters are periodically replicated to generate stable Q' values to suppress training fluctuations. After random sampling from the experience replay buffer, the loss function value is calculated by combining it with the optimal action of the next state, thus completing the behavior network. Parameter optimization.
[0077] like Figure 4 As shown, Figure 4 This is a flowchart of a Markov decision process, specifically including: obtaining pixel values within the agent's field of view collected at four consecutive time steps from the field of view buffer, thereby obtaining the agent's state. The agent's state is input into the behavior network to obtain the corresponding action to be performed at the current moment. After executing the action, it determines whether the preset oscillation condition is met. If not, it continues to interact with the environment, uses the behavioral network to decide the agent's next action, and calculates the reward value. If the conditions are met, the key point detection results will be output.
[0078] To verify the effectiveness of the method provided in this application, taking a medical image as the target image as an example, key point detection is performed using the method provided in this invention, such as... Figure 5 As shown, Figure 5 This is a diagram showing the results of keypoint detection across multiple images. Figure 5 Figures (a)-(k) are comparison images of key points obtained using the method provided in this invention and the actual key points. Figure 5 The red dots represent detected intelligent agents, and the bright green dots represent key target points.
[0079] The multi-agent and environment interaction learning method for image keypoint detection provided by this invention specifically includes the following aspects: 1. A multi-agent and environment interaction learning method for image key point detection is proposed, which integrates Markov decision process, multi-scale search strategy and multi-agent cooperation mechanism to achieve more efficient and accurate medical image key point localization.
[0080] 2. A Markov decision process design adapted to image keypoint detection is proposed. The pixel values of the agent's field of vision at four consecutive time steps are used as the state, and the Euclidean distance change is used as the reward function to achieve accurate interaction between the agent and the image environment. Furthermore, a multi-scale search strategy for the agent is combined with the field of vision and the movement step size from large to small, and the oscillation state is used as the adaptive switching condition, which takes into account both detection efficiency and positioning accuracy.
[0081] 3. Implicit and explicit communication of the behavioral networks of each agent are proposed. The behavioral networks of all agents share the same set of convolutional and pooling layer parameters. The output features of the previous fully connected layer of each agent are concatenated with the mean feature vector of the output of the previous fully connected layer of all agents, and used as the input of the next fully connected layer. This allows multiple agents to share information and complement each other's perspectives, thereby improving the accuracy of action prediction.
[0082] 4. The loss function proposed in this invention separates "action selection" from "value calculation", which can accurately evaluate the difference between the predicted Q value output by the current behavior network and the target Q' value calculated by the target network.
[0083] This method can capture key point features in medical images more fully by interacting with the environment through multiple agents and combining multi-scale search strategies. In detection scenarios such as skeletal anatomical points and organ boundary points, the accuracy and detection efficiency of key point localization are significantly improved compared with existing methods. At the same time, the robustness is also effectively enhanced in medical images with complex textures and overlapping tissues.
[0084] When applying the image keypoint detection method based on multi-agent and environment interaction learning provided in this manual, it is not necessary to... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this manual does not impose any restrictions on it.
[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
Claims
1. A multi-agent interaction learning method for image keypoint detection, characterized in that, The target image is the environment of the agents, where each agent represents a movable point on the target image, and the target image includes multiple agents; the method includes: Obtain the current state of each agent; the state of an agent includes the pixel values of its field of view in the target image at multiple consecutive time points; For any agent, the action of the agent is decided by the behavior network based on the agent's current state, and the action to be taken at the current moment is determined. The position of the agent at the next moment is determined based on the action and the step size. The behavior network is the agent action decision module in the Markov decision process, which is responsible for the interaction learning between the agent and the environment. Its network parameters are iteratively optimized through reinforcement learning training. If the agent has the same position at multiple moments within a consecutive preset time period, the agent's current scale search level is adjusted to the next scale search level. The agent's state at the next moment is determined based on the agent's position at the next moment and the adjusted field of view. Each scale search level includes the field of view and the movement step size. The field of view and movement step size of the previous scale search level are both greater than those of the next scale search level. If the agent's position is not the same at multiple moments within a consecutive preset time period, the state of the agent at the next moment is determined based on the agent's position at the next moment and the current field of vision. Based on the agent's state at the next moment, the action of the agent is decided by the behavior network until, at the minimum scale search level, the agent's position is the same at multiple moments within a consecutive preset time period. The final positions of all agents are determined as key points on the target image.
2. The method according to claim 1, characterized in that, The behavioral network consists of convolutional layers, pooling layers, and fully connected layers. All agents share the same set of convolutional and pooling layer network parameters to extract general features of the image. Each agent has its own independent set of fully connected layer parameters, and there is an interactive communication mechanism between the fully connected layers of different agents. The specific implementation process of the interactive communication mechanism is as follows: For any agent's adjacent upper and lower fully connected layers, obtain the output features of the upper fully connected layer of the current agent, and simultaneously obtain the output features of the upper fully connected layers of other agents at the same level, and calculate the mean feature vector of the output features of all upper fully connected layers; concatenate the mean feature vector with the output features of the upper fully connected layer of the current agent along the channel dimension; use the concatenated feature vector as the input feature of the lower fully connected layer of the current agent.
3. The method according to claim 1, characterized in that, Markov decision processes include a set of states, a set of actions, a reward function, and a reward discount factor. The state set includes: selecting pixel values corresponding to the agent's field of vision at multiple consecutive time points, stitching them together along the channel dimension, and using the stitched feature map as the agent's current state; The action set includes: designing the agent's actions as discrete two-dimensional movement actions, including along... Move along the positive axis Move along the negative axis Move along the positive axis Move in the negative direction of the axis; The reward function includes: ； in, The reward value for the agent. Perform actions for the agent Distance between the front and the target key point Perform actions for the agent The distance from the target key point For reward scaling factor; Reward discount factors include: , Used to balance the weight of current rewards and future rewards in cumulative rewards: when When =0, the cumulative reward is related to the current reward; when When the value is 1, the current reward and the future reward have equal weight in the cumulative reward.
4. The method according to claim 1, characterized in that, The training process of a behavioral network includes: The initial behavior network is trained by drawing multiple sets of experience samples from the experience replay buffer, and its parameters are iteratively updated. The initial behavior network after training is defined as the behavior network. During training, a multi-scale search strategy is employed to alter the field of vision and step size of the sample agents, changing them from a large field of vision and large step size to a small field of vision and small step size; at each interval... N The next step is to copy the updated parameters of the initial behavioral network to the target network to update the parameters of the target network. The target network and the behavior network have the same structure, but their parameter update logic is independent of each other.
5. The method according to claim 4, characterized in that, Loss function used during behavioral network training for: ; in, As a reward discount factor, Selecting behavioral networks Moment State action , Calculate actions for the target network value, Calculate the initial behavior network Moment Action value, To reward scaling factor, Perform actions for the agent Distance between the front and the target key point Perform actions for the agent The distance between the target key point and the target.
6. The method according to claim 4, characterized in that, The method further includes the following steps before training the initial behavior network: The initial sample state is obtained by acquiring the pixel values of the sample agent's field of view at multiple consecutive time points in the sample image; there are multiple sample agents. Based on the initial sample state, the initial behavior network is used to make decisions on the actions of the sample agent and determine the actions of the sample agent. Execute the action and movement step of the sample agent to obtain the position of the sample agent at the next moment; Using Markov decision processes, the reward value is calculated based on the initial position and next time step position of the sample agent, as well as the position of the corresponding key point of the sample. The next sample state is obtained and the quadruple is placed in the experience replay buffer. The quadruple includes the initial sample state, action, reward value and next sample state of the sample agent at the current time step. Based on the next sample state of the sample agent, continue the search to obtain a new sample state and generate a new quadruple, which is then stored in the experience replay cache.
7. The method according to claim 6, characterized in that, The method further includes: During the parameter update of the initial behavior network, the updated initial behavior network parameters are used simultaneously to recalculate the predicted action corresponding to the sample state in the experience replay buffer, construct an updated quadruple containing the predicted action, and replace the corresponding original quadruple in the experience replay buffer with the updated quadruple.