A circular reconstruction system based on reinforcement learning

Abstract

This invention relates to the field of circular reconstruction algorithms and discloses a circular reconstruction system based on reinforcement learning. This system uses Markov decision processes to describe the reconstruction process of multiple circular arcs; that is, estimating the parameters of one arc corresponds to making one decision, and estimating the parameters of multiple arcs corresponds to making multiple decisions. The invention receives actions output by the agent module through an environment module, calculates the reward generated by the action using an average measure estimator, changes the environment state based on the action, and feeds back the generated reward and the changed state to the agent. The agent module receives the reward and state output by the environment module, generates new actions and feeds them back to the environment module, and uses the PPO reinforcement learning algorithm to optimize and update the agent's action policy. A recording module records the optimal scene segment during the interaction between the agent and the environment, containing each action. This system benefits from the high robustness of the average measure estimator and the high stability of the PPO algorithm.

Description

Technical Field

[0001] This invention relates to the field of circular reconstruction algorithm technology, specifically a circular reconstruction system based on reinforcement learning. Background Technology

[0002] Circles are ubiquitous in our lives; therefore, circle detection is not only a fundamental task in image processing research but also a widely applied technology in many fields. For example, in the military, drones and missiles use circle reconstruction technology to locate targets such as oil tanks and radomes. In industry, circle reconstruction methods are used to locate industrial products or reconstruct circular defects. In the transportation sector, circle reconstruction is often used to assist autonomous driving systems in recognizing traffic signs. In the medical field, it is frequently used to measure pupil position and for medical imaging analysis. Therefore, research on circle reconstruction algorithms has significant positive implications for national development and people's livelihoods.

[0003] The current mainstream circle reconstruction algorithms are as follows: methods based on the geometric properties of circles, such as tangency or symmetry; area-based methods; Hough transform-based methods; and convolutional neural network-based circle reconstruction methods. The first type of algorithm relies on gradient features and low-level edge information. However, in reality, edge maps often contain a large amount of edge information that is not part of the target circle. Furthermore, these algorithms struggle to separate single-circle edges from multiple circle edges, instead treating them as a whole for subsequent algorithms. Therefore, they are sensitive to noise and occlusion when the image background is complex and contains a lot of noise, and the reconstruction accuracy is very low. The second and third types of algorithms have improved robustness compared to the first type but lack flexibility and cannot reconstruct circles with large eccentricities. However, circles in real life are often imperfect, limiting the application scenarios of these algorithms. The third type of algorithm has improved robustness compared to the first two types, but it relies on learning from the dataset and selecting candidate boxes for potential circular targets. It is a specific type of circle reconstructor and not easily used for multi-circle detection. Therefore, its application scenarios are not as wide as the first type of algorithm. Thus, designing a robust multi-circle detection method is a key focus of current research on circle reconstruction and other object detection algorithms. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a circular reconstruction system based on reinforcement learning. The core idea is that estimating the parameters of one circular arc corresponds to making one decision, and estimating the parameters of multiple circular arcs corresponds to making multiple decisions. Therefore, a Markov Decision Process (MDP) can be used to describe the reconstruction process of multiple circular arcs. A MDP refers to the interaction process between an agent and its environment. This invention receives actions (i.e., decisions) output by the agent module through an environment module, calculates the reward generated by the action using an average measure estimator, changes the environment state based on the action, and feeds back the generated reward and the changed state to the agent. The agent module receives the reward and state output by the environment module, generates new actions and feeds them back to the environment module, and uses the PPO reinforcement learning algorithm to optimize and update the agent's action policy. A recording module records the optimal scene segment during the interaction between the agent and the environment. The actions contained in the optimal scene segment represent the circular arcs reconstructed by this invention. Thanks to the high robustness of the average measure estimator and the high stability of the PPO algorithm, this invention can stably and robustly reconstruct multiple circles from given image data, solving the problem of multi-circle reconstruction.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: a circular reconstruction system based on reinforcement learning, comprising an agent module, an environment module, and a recording module;

[0008] The environment module receives actions output by the agent module, calculates the reward generated by the action, changes the environment state according to the action, and then feeds back the generated reward and the changed state to the agent. For ease of description, the time intervals are referred to below. At that time, the state of the environment is The received action is The resulting reward is The transformed environmental state is in For state space, This is the action space.

[0009] The agent module receives rewards and states from the environment module, generates new actions, feeds these new actions back to the environment module, and optimizes and updates the agent's policy function.

[0010] The recording module is used to record the interaction process between the agent module and the environment module. The environment starts from an initial state and, after several interactions with the agent, reaches a termination state. This process from the initial state to the termination state is called an episode. When an episode ends, the environment is reset to its initial state, the agent's accumulated reward is cleared, time t is also cleared, and then the next episode begins. Each episode has a corresponding accumulated reward. The episode corresponding to the highest accumulated reward is called the optimal episode. The optimal episode represents the result reconstructed by this circular reconstruction system.

[0011] Furthermore, the environment module includes a data point set generation module, a model point set generation module, a point set similarity estimation module, and a state transition module.

[0012] The data point set generation module extracts the edges of a given image using an edge detection operator, obtaining all edge points of the image. These edge points are the data point set D used to reconstruct the circle.

[0013] The model point set generation module is used to receive the action a output by the agent. t , will action a t The parameter β is analyzed as a single circular arc. t According to parameter β t The corresponding single arc point set M can be obtained. t The parameter vector β of the arc contains five parameters: the central angle of the arc, the deflection angle of the starting point of the arc, the radius, the x-coordinate of the center, and the y-coordinate of the center. The action vector a is also a 5-dimensional vector, with its five components corresponding to the five parameters of the arc. It can be seen that one action vector generates a set of arc points, and the sets of arc points generated by multiple actions combined form a union of arc point sets. In the current scene, action a... t and the union of the arc points generated by all actions preceding this action. This is called the model point set.

[0014] The point set similarity estimation module is used to calculate the similarity of action a. t The generated model point set The similarity between the data point set D and the data point set D is used to calculate action a. t The corresponding reward. Let's remember the model point set. The similarity between the data point set D and the data point set D is: Action a t The corresponding reward r t+1 By a t The generated model point set And the previous action a t-1 The generated model point set The calculation formula is determined by mutual agreement and is as follows:

[0015] The state transition module is used to calculate the state transition caused by action a. t The resulting environmental state transitions. The design of the environmental state is crucial to this circular reconstruction system. Therefore, this system presupposes a maximum number of interactions between the agent and the environment in each scene as w, i.e., 0 ≤ t ≤ w. Simultaneously, the value range of each component of the action vector a is set to [-1, 1]. Let's denote the action vector where each component reaches its minimum value as ai. L The action vector when each component reaches its maximum value is a. U The superscript L indicates the lower bound of the value, and the superscript U indicates the upper bound of the value. Based on the above explanation, the initial state of this system, i.e., the state at time t=0, is defined as s0 = (a L ,a L ,…,a L ), where a L The number of elements is w-1, and the state s at time t is... t =(a0,a1,…,a t-1 ,a L ,…,a L ), where a L The number of elements is wt-1. Execute action a. t Afterwards, the state will change to s. t+1 =(a0,a1,…,a t ,a L ,…,a L ), where a L The number of elements is wt-2. It is worth noting that when the stopping condition for the split is met, the state of the environment will be reset, that is, the state will change to s0. The stopping condition for the split in this invention is that if the length of the arc generated at the current moment is less than the preset length threshold, or the distance between the arc generated at the current moment and the model point set generated at the previous moment is less than the preset distance threshold, or the number of interactions between the agent and the environment reaches the preset maximum number, the state of the environment will be reset, that is, the state will change to s0.

[0016] Preferably, the agent module employs an agent based on a proximal policy optimization algorithm, which comprises two neural networks: an actor network and a critic network. The actor network is the agent's policy function, generating new actions based on the state output by the environment module. The parameters θ of the actor network and the critic network are determined based on the state and reward output by the environment module. All of these will be updated.

[0017] Preferably, the data point set generation module extracts the edges of a given image using the Canny edge detection operator, comprising the following four steps:

[0018] 1. Use a Gaussian filter to remove noise from the image;

[0019] 2. Calculate the gradient magnitude and direction for each pixel in the image;

[0020] Third, apply nonmaximum suppression to eliminate stray responses caused by edge detection;

[0021] Fourth, use dual thresholding to determine strong edges and remove false edges. The image edge point set obtained after the above steps is the data point set D used to reconstruct the circle.

[0022] Preferably, the point set similarity estimation module uses an average measure estimator to estimate the similarity between the model point set and the data point set. The formula for calculating the average measure is as follows:

[0023]

[0024] In the formula, |M| is the measure of model M, where the model is an arc and the measure of the arc is its length. ∈ is a very small positive number used to prevent the denominator from being 0. d is the error between the model point set and the data point set, called the model error. λ is used to adjust the weights of the measure and the model error, and is fixed at 2. The model error d is defined as follows:

[0025]

[0026] In the formula, M δ Let |M| represent the set of points uniformly sampled from the arc M at intervals of δ. δ | represents the point set M δ The number of points contained, where ||pq|| represents the Euclidean distance between points p and q.

[0027] Compared with existing technologies, this invention provides a circular reconstruction system based on reinforcement learning, which has the following beneficial effects:

[0028] This invention employs Markov Decision Processes (MDPs) to describe the reconstruction process of multiple circular arcs. The core idea is that estimating the parameters of one circular arc corresponds to making one decision, and estimating the parameters of multiple arcs corresponds to making multiple decisions. A Markov Decision Process refers to the interaction process between an agent and its environment. This invention receives actions (i.e., decisions) output by the agent module through an environment module, calculates the reward generated by the action using an average measure estimator, changes the environment state based on the action, and feeds back the generated reward and the changed state to the agent. The agent module receives the reward and state output by the environment module, generates new actions and feeds them back to the environment module, and uses the PPO reinforcement learning algorithm to optimize and update the agent's action policy. A recording module records the optimal scene segment during the interaction between the agent and the environment. The actions contained in the optimal scene segment represent the circular arcs reconstructed by this invention. Thanks to the high robustness of the average measure estimator and the high stability of the PPO algorithm, this invention can stably and robustly reconstruct multiple circles from given image data. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the system flow of the present invention;

[0030] Figure 2 This is a flowchart of the system structure of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] To address the issue that current multi-circle detection algorithms lack robustness in detecting circles, resulting in low accuracy, a circle reconstruction system based on reinforcement learning is proposed. (See attached image.) Figure 1 The system mainly consists of an environment module and an intelligent agent module, wherein:

[0033] The environment module receives actions output by the agent module, calculates the reward generated by the action, changes the environment state based on the action, and then feeds back the generated reward and the changed state to the agent. For example... Figure 2 As shown, the environment module includes a data point set generation module, a model point set generation module, a point set similarity estimation module, and a state transition module, wherein:

[0034] The data point set generation module extracts edges from a given image using the Canny edge detection operator, comprising the following four steps: 1. Removing noise from the image using a Gaussian filter; 2. Calculating the gradient magnitude and direction of each pixel in the image; 3. Applying non-maximum suppression to eliminate stray responses from edge detection; 4. Using a double threshold to determine strong edges and remove false edges. The image edge point set obtained after these steps is the data point set D used to reconstruct the circle.

[0035] The model point set generation module is used to receive the action a output by the agent. t , will action a t The parameter β is analyzed as a single circular arc. t According to parameter β t The corresponding single arc point set M can be obtained. t The parameter vector β of the arc contains five parameters: the central angle of the arc, the deflection angle of the starting point of the arc, the radius, the x-coordinate of the center, and the y-coordinate of the center. The action vector a is also a 5-dimensional vector, with its five components corresponding to the five parameters of the arc. It can be seen that one action vector generates a set of arc points, and the sets of arc points generated by multiple actions combined form a union of arc point sets. In the current scene, action a... t and the union of the arc points generated by all actions preceding this action. This is called the model point set.

[0036] This invention resolves action a into the parameter β of a single circular arc as follows: The value range of each component of the action vector a is set to [-1, 1]. The value ranges of each component of the circular arc parameter β are as follows: the central angle and the deflection angle of the arc's starting point are both [0, 360°]; the minimum value of the radius is 0, and the maximum value is set according to the actual situation; the maximum and minimum values ​​of the horizontal and vertical coordinates of the center can be set based on the bounding box of the data point set. Therefore, the MinMax scaling method can resolve action a into the parameter β of a single circular arc.

[0037] The point set similarity estimation module is used to calculate the similarity of action a. t The generated model point set The similarity between the data point set D and the data point set D is used to calculate action a. t The corresponding reward. Let's remember the model point set. The similarity between the data point set D and the data point set D is: Action a t The corresponding reward r t+1 By a t The generated model point set And the previous action a t-1 The generated model point set The calculation formula is determined by mutual agreement and is as follows:

[0038] Preferably, the point set similarity estimation module uses an average measure estimator to estimate the similarity between the model point set and the data point set. The formula for calculating the average measure is as follows:

[0039]

[0040] In the formula, |M| represents the measure of model M. In this paper, the model is an arc, and the measure of the arc is its length. ∈ is a very small positive number used to prevent the denominator from being zero. d represents the error between the model point set and the data point set, called the model error. λ is used to adjust the weights of the measure and the model error; in this system, it is fixed at 2. The model error d is defined as follows:

[0041]

[0042] In the formula, M δ This represents the set of points uniformly sampled from the arc M at intervals of δ. |M δ | represents the point set M δ The number of points contained. ||pq|| represents the Euclidean distance between points p and q.

[0043] The state transition module is used to calculate the state transition caused by action a. t The resulting environmental state transitions. The design of the environmental state is crucial to this circular reconstruction system. Therefore, this system presupposes a maximum number of interactions between the agent and the environment in each scene as w, i.e., 0 ≤ t ≤ w. Simultaneously, the value range of each component of the action vector a is set to [-1, 1]. Let's denote the action vector where each component reaches its minimum value as ai. L The action vector when each component reaches its maximum value is a. U The superscript L indicates the lower bound of the value, and the superscript U indicates the upper bound of the value. Based on the above explanation, the initial state of this system, i.e., the state at time t=0, is defined as s0 = (a L ,a L ,…,a L ), where a L The number of elements is w-1. The state s at time t... t =(a0,a1,…,a t-1 ,a L ,…,a L ), where a L The number of elements is wt-1. Execute action a. t Afterwards, the state will change to s. t+1 =(a0,a1,…,a t ,a L ,…,a L), where a L The number of elements is wt-2. It is worth noting that when the scene-splitting stopping condition is met, the environment state will be reset, i.e., the state will change to s0. In this embodiment, the scene-splitting stopping condition is set as follows: if the length of the arc generated at the current moment is less than a preset length threshold, or the distance between the arc generated at the current moment and the model point set generated at the previous moment is less than a preset distance threshold, or the number of interactions between the agent and the environment reaches a preset maximum number, the environment state will be reset, i.e., the state will change to s0.

[0044] The agent module employs an agent based on the Proximal Policy Optimization (PPO) algorithm. The PPO algorithm comprises two neural networks: an actor network and a critic network. The actor network serves as the agent's policy function, generating new actions based on the state output by the environment module. The parameters θ of the actor network and the critic network are determined based on the state and reward output by the environment module. All will be updated. The main process of the PPO algorithm is as follows:

[0045] I. Initialize the actor network (actor) with parameters θ0 and the critic network (critic) with parameters θ0 and θ0 respectively. And set hyperparameters, such as the learning rate;

[0046] 2. To perform the (k+1)th update of the actor network and the critic network (k≥0), we need to utilize the actor network actor(θ). k The interaction occurs n times with the environment, resulting in the interaction trajectories (s0, a0, r1), (s1, a1, r2), ..., (s n-1 ,a n-1 ,r n ), and calculate the dominance function A0, A1, ..., A n The formula for calculating the dominance function is as follows:

[0047]

[0048] in That is, the network of critics The output value.

[0049] III. Based on the following loss function, the parameters θ of the actor network and the parameters of the critic network are... Update:

[0050]

[0051]

[0052]

[0053] c i (θ)=max(min(b i (θ), 1+ε), 1-ε)

[0054] Where, p θ (a|s) represents the probability of taking action a given state s when the policy is actor(θ), and ε is the clipping factor, which is usually taken as ε = 2.

[0055] IV. Steps 2 and 3 of the iterative loop.

[0056] The recording module is used to record the optimal scene segmentation during the interaction between the agent and the environment. Each action contained in the optimal scene segment represents the arcs reconstructed by this invention. Thanks to the high robustness of the average metric estimator and the high stability of the PPO algorithm, this invention can stably and robustly reconstruct multiple circles from given image data.

[0057] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A circular reconstruction system based on reinforcement learning, characterized in that: It includes an environment module, an intelligent agent module, and a recording module; The environment module receives actions output by the agent module, calculates the reward generated by the action, changes the environment state according to the action, and then feeds back the generated reward and the changed state to the agent at time [time missing]. At that time, the state of the environment is The received action is The resulting reward is The transformed environmental state is in For state space, For action space; The agent module is used to receive the rewards and status output by the environment module, generate new actions, feed the new actions back to the environment module, and optimize and update the agent's policy function. The recording module is used to record the interaction process between the agent module and the environment module. The environment starts from the initial state and will reach the termination state after several interactions with the agent. This process from the initial state to the termination state is called a segment. When a segment ends, the environment will be reset to the initial state, the cumulative reward obtained by the agent will be cleared, and time t will also be cleared. Then the next segment will begin. Each segment has a corresponding cumulative reward. The segment corresponding to the highest cumulative reward is called the best segment. The best segment represents the result reconstructed by this circular reconstruction system.

2. The circular reconstruction system based on reinforcement learning according to claim 1, characterized in that: The environment module includes a data point set generation module, a model point set generation module, a point set similarity estimation module, and a state transition module.

3. The circular reconstruction system based on reinforcement learning according to claim 1, characterized in that: The agent module employs an agent based on a proximal policy optimization algorithm. This algorithm comprises two neural networks: an actor network and a critic network. The actor network is the agent's policy function, which generates new actions based on the state output by the environment module. The parameters of both the actor network and the critic network are updated based on the state and reward output by the environment module.

4. A circular reconstruction system based on reinforcement learning according to claim 2, characterized in that: The data point set generation module extracts the edges of a given image using the Canny edge detection operator, and includes the following four steps:

1. Use a Gaussian filter to remove noise from the image; 2. Calculate the gradient magnitude and direction for each pixel in the image; Third, apply nonmaximum suppression to eliminate stray responses caused by edge detection; Fourth, use dual thresholds to determine strong edges and remove false edges. The image edge point set obtained after the above steps is the data point set D used to reconstruct the circle.

5. A circular reconstruction system based on reinforcement learning according to claim 2, characterized in that: The model point set generation module is used to receive the action a output by the agent. t , will action a t The parameter β is analyzed as a single circular arc. t According to parameter β t The corresponding single arc point set M can be obtained. t The parameter vector β of the arc contains five parameters: the central angle of the arc, the deflection angle of the starting point of the arc, the radius, the x-coordinate of the center, and the y-coordinate of the center. The action vector a is also a 5-dimensional vector, with its five components corresponding to the five parameters of the arc. One action vector generates a set of arc points, and the sets of arc points generated by multiple actions are combined to form a union of arc point sets. In the current scene, action a... t and the union of the arc points generated by all actions preceding this action. This is called the model point set.

6. A circular reconstruction system based on reinforcement learning according to claim 2, characterized in that: The point set similarity estimation module is used to calculate the similarity of action a. t The generated model point set The similarity between the data point set D and the data point set D is used to calculate action a. t The corresponding rewards can be recorded as model point sets. The similarity between the data point set D and the data point set D is: Action a t The corresponding reward r t+1 By a t The generated model point set And the previous action a t-1 The generated model point set The calculation formula is determined by mutual agreement and is as follows:

7. A circular reconstruction system based on reinforcement learning according to claim 2, characterized in that: The state transition module is used to calculate the state transition caused by action a. t The environmental state transitions caused by this are pre-defined as the maximum number of interactions between the agent and the environment in each scene, where w is the maximum number of interactions, i.e., 0 ≤ t ≤ w. Simultaneously, the value range of each component of the action vector a is set to [-1, 1]. The action vector where each component reaches its minimum value is denoted as a0. L The action vector when each component reaches its maximum value is a. U The superscript L indicates the lower bound of the value, and the superscript U indicates the upper bound of the value. The initial state, i.e., the state at time t=0, is defined as s0=(a L ,a L ,…,a L ), where a L The number of elements is w-1, and the state s at time t is... t =(a0,a1,…,a t-1 ,a L ,…,a L ), where a L The number of items is wt-1, and action a is executed. t Afterwards, the state will change to s t+1 =(a0,a1,…,a t ,a L ,…,a L ), where a L The number of elements is wt-2. When the stopping condition for the split is met, the state of the environment will be reset, that is, the state will change to s0. The set stopping condition for the split is that the length of the arc generated at the current moment is less than the preset length threshold, or the distance between the arc generated at the current moment and the model point set generated at the previous moment is less than the preset distance threshold, or the number of interactions between the agent and the environment reaches the preset maximum number. The state of the environment will be reset, that is, the state will change to s0.

8. A circular reconstruction system based on reinforcement learning according to claim 6, characterized in that: The point set similarity estimation module uses an average measure estimator to estimate the similarity between the model point set and the data point set. The formula for calculating the average measure is as follows: In the formula, |M| is the measure of model M, the model is an arc, and the measure of the arc is its length. ∈ is a positive number to prevent the denominator from being 0. d is the error between the model point set and the data point set, called the model error. λ is used to adjust the weights of the measure and the model error, and is fixed at 2. The model error d is defined as follows: In the formula, M δ Let |M| represent the set of points uniformly sampled from the arc M at intervals of δ. δ | represents the point set M δ The number of points contained, where ||pq|| represents the Euclidean distance between points p and q.

Images