An automatic three-coordinate measurement method based on deep reinforcement learning

Abstract

The application provides an automatic three-coordinate measurement method based on deep reinforcement learning, and belongs to the technical field of computer systems based on specific calculation models, and comprises an agent deep learning model training step and an online measurement step, the agent deep learning model training step comprising: constructing a state vector; designing an agent action space; constructing an end-to-end agent neural network architecture; designing a reinforcement learning reward mechanism; training a hybrid action agent; the online measurement step comprising: constructing a state vector and inputting the state vector into a position predictor to find an action with the maximum expected return estimation; controlling a measurement needle to perform the action and deleting corresponding to-be-measured feature geometric information of the measurement needle; and updating the state vector according to an execution result of the foregoing steps. The automatic three-coordinate measurement method based on deep reinforcement learning can quickly adapt and balance path safety and efficiency, can avoid problems such as redundancy and collision, does not need manual intervention, and realizes automatic adaptation and rapid efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-coordinate measurement, and particularly relates to an automatic three-coordinate measurement method based on deep reinforcement learning. BACKGROUND

[0002] A three-coordinate measurement system (CMM) acquires three-dimensional coordinate points of an object surface through a probe contact or a non-contact sensor, and is used for size detection, reverse engineering and quality control.

[0003] A traditional three-coordinate measurement process relies on manual programming or fixed measurement paths, and needs to pre-set a moving track, an angle of a probe and a measurement sequence of a feature to be measured. For a complex workpiece with multiple types of geometric features or irregular geometric features, manual programming is inefficient, and it is difficult to balance efficiency and obstacle avoidance safety. In addition, the traditional measurement method relies on manual programming and cannot dynamically adapt to different workpieces to be measured, resulting in rigid measurement processes and difficulty in meeting the flexible needs of an automatic production line.

[0004] In addition, there is a measurement path planning method based on a heuristic algorithm. This method has the following problems: 1. The representation of the workpiece to be measured is not flexible enough, and the algorithm has limited generalization ability, which cannot adapt to complex and variable workpieces; 2. Most algorithms only plan the measurement sequence of the feature to be measured, and do not plan the specific moving path of the probe and the obstacle avoidance strategy; 3. The heuristic algorithm has a large amount of calculation and a long optimization time, and the feasibility in actual use is not high. SUMMARY

[0005] To solve the problems of the existing three-coordinate measurement fixed measurement path or the measurement path planning method based on the heuristic algorithm, the present application proposes an automatic three-coordinate measurement method based on deep reinforcement learning, which can solve the above problems.

[0006] To solve the above technical problems, the present application adopts the following technical solutions:

[0007] An automatic three-coordinate measurement method based on deep reinforcement learning includes an agent deep learning model training step and an online measurement step, and the agent deep learning model training step includes:

[0008] Constructing a state vector , the state vector includes geometric information of a feature to be measured, probe information and safety plane information;

[0009] Designing an agent action space A, including a first-level action space and a second-level action space , the first-level action space has a plurality of actions, and the second-level action space has parameter information corresponding to each action respectively;

[0010] constructing an end-to-end agent neural network architecture, including a position predictor and an action selector ;

[0011] designing a reinforcement learning reward mechanism, defining various situations that may occur during the execution of the measurement task by the agent and designing corresponding reward signals r;

[0012] training a hybrid action agent based on parameterized Q-Learning, including: obtaining the state observation at the current time from the simulator and inputting it to the agent, the agent outputting the measurement needle action , the simulator executing and obtaining the next state and the corresponding reward , while outputting an indication of whether the interaction is over , calculating the expected return estimate of the action that can obtain the maximum cumulative reward from the current time to the time when the interaction terminates , calculating the loss function according to , and updating the position predictor and the action selector according to the loss function;

[0013] the online measurement step includes:

[0014] (21) obtaining the to-be-measured feature geometric information from the workpiece design drawing, obtaining the measurement needle information and the safety plane information from the three-coordinate measurement, and constructing a state vector ;

[0015] (22) inputting to the position predictor, the position predictor outputting the moving position according to the state vector in order , the action selector finding all possible actions of the measurement needle moving from the current state to the moving position, and calculating the expected return estimate of each action, and finding the action with the maximum expected return estimate ;

[0016] (23) controlling the measurement needle to execute the action and deleting the to-be-measured feature geometric information corresponding to it;

[0017] (24) updating the state vector according to the execution result of step (23), and repeating steps (22)-(23) until the termination condition is met to terminate the execution.

[0018] In some embodiments, the state vector is constructed as In the step, the to-be-measured feature geometric information, the probe information and the safety plane information directly obtained from the workpiece design drawing are original state information ;

[0019] Constructing a state vector The step also includes constructing an embedding model library The obtained original state information is mapped into a dimensionally aligned state vector, which is a state vector ;

[0020] ;

[0021] Wherein, d is a fixed state dimension, and N is the number of vectors constituting the state.

[0022] In some embodiments, in the construction of an end-to-end agent neural network architecture, a position predictor is used to predict the moving position of the probe according to the state vector , which is a moving position prediction value ;

[0023] ;

[0024] Wherein, ;

[0025] An action selector is used to find all possible actions of the probe from the agent action space A according to the state vector and the moving position prediction value , and calculate the expected return estimate of each action:

[0026] .

[0027] In some embodiments, in the step of designing a reinforcement learning reward mechanism, the reward signal r includes: completing the measurement of all features ; completing the measurement of a single feature ; probe collision ; movement ; rotation and illegal action , wherein each reward signal satisfies:

[0028] ;

[0029] is negatively correlated with the moving distance.

[0030] In some embodiments, in the hybrid action agent training step based on parameterized Q-Learning, the estimation of the maximum cumulative discounted reward that the agent can obtain from time t to the end of the interaction at time T in the interaction at time t is:

[0031] ;

[0032] wherein, is the discount factor, is the reward at time t, d t is an indication of whether the interaction at time t is over, is the estimation of the cumulative discounted reward that can be obtained in the future for all possible actions a' at time t+1.

[0033] In some embodiments, in the hybrid action agent training step based on parameterized Q-Learning, the loss function is calculated as:

[0034] ;

[0035] wherein, θ is the policy network parameter, is the cumulative discounted reward estimation function based on the current policy network parameter θ, and are the expectations of the value of in the action space A and the experience buffer D, respectively.

[0036] In some embodiments, in step (23), if the action of is moving or rotating, the method further comprises updating the needle information and the safety plane information.

[0037] Compared with the prior art, the automatic three-coordinate measurement method based on deep reinforcement learning has the advantages and positive effects that: the automatic three-coordinate measurement method based on deep reinforcement learning of the application can quickly adapt and balance path safety and efficiency, and can avoid problems such as redundancy or collision when facing complex workpieces containing multiple types of features, by obtaining feature geometric information to be measured from a design drawing of a workpiece to be measured, obtaining stylus information and safety plane information through a three-coordinate measuring machine measurement needle state, and constructing a state vector according to the above. When the workpiece is replaced, only the above information needs to be imported again, the agent has generalization ability for complex and variable workpieces to be measured in the actual production process, and does not need manual intervention, realizes automatic adaptation, and is fast and efficient. The reinforcement learning agent (Agent) based on the Transformer neural network architecture is constructed in combination with the characteristics of the measurement task, the reward (Reward) signal is designed in combination with the actual measurement task target, and the agent is trained through reinforcement learning, so that the three-coordinate measuring machine finally realizes self-adaptive workpiece recognition, measurement path planning and stylus obstacle avoidance functions in the complex workpiece measurement process, and is suitable for scenes such as mechanical manufacturing, precision machining, quality detection and other scenes that need high-precision geometric dimension measurement.

[0038] Other features and advantages of the application will become more apparent after reading the detailed description of the embodiments of the application in combination with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is an agent action space diagram defined in an embodiment of the automatic three-coordinate measurement method based on deep reinforcement learning proposed by the application;

[0040] Figure 2 is an overall flow design schematic diagram of an automatic three-coordinate measurement agent in an embodiment of the automatic three-coordinate measurement method based on deep reinforcement learning proposed by the application;

[0041] Figure 3 is a curve diagram of the average reward of the agent obtained by the agent in an embodiment of the automatic three-coordinate measurement method based on deep reinforcement learning proposed by the application with the change of the training step number;

[0042] Figure 4 is a curve diagram of the measurement task completion rate of the agent in an embodiment of the automatic three-coordinate measurement method based on deep reinforcement learning proposed by the application with the change of the training step number. DETAILED DESCRIPTION

[0043] The specific embodiments of the application will be further described in detail below in combination with the accompanying drawings.

[0044] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] It should be noted that, in the description of the present application, the terms indicating the direction or position relationship of "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like are based on the direction or position relationship shown in the drawings, which is only for the convenience of description, and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited.

[0046] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection", "fixing" and the like should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship of two elements. For a person of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0047] Embodiment one, see Figure 2 As shown in the drawings, the present embodiment proposes an automatic three-coordinate measurement method based on deep reinforcement learning, which includes an agent deep learning model training step and an online measurement step. The agent deep learning model training step includes:

[0048] Constructing a state vector The state vector includes the measured feature geometric information, the probe information and the safety plane information. By obtaining the measured feature geometric information from the design drawing of the measured workpiece, obtaining the probe information and the safety plane information through the probe state of the three-coordinate measuring machine, and constructing the state vector according to the above, when facing a complex workpiece containing multiple types of features, the path safety and efficiency can be quickly adapted and balanced, and the problems of redundancy or collision can be avoided. When the workpiece is replaced, only the above information needs to be imported again, the agent has the generalization ability to the complex and changeable measured workpiece in the actual production process, and does not need manual intervention, realizes automatic adaptation, and is fast and efficient.

[0049] designing an agent action space A, including a primary action space and a secondary action space , the primary action space having a plurality of actions, and the secondary action space having parameter information corresponding to each action respectively. By designing the agent action space A to include a primary action space and a secondary action space , the action and the parameter information corresponding to the action are saved separately, and are combined for use by taking values from the two action spaces respectively, so that a small amount of storage space can cover a variety of combination schemes.

[0050] constructing an end-to-end agent neural network architecture, including a position predictor and an action selector . By combining the probe movement mode of an actual three-coordinate measuring machine, an end-to-end hybrid action space reasoning framework is designed. After training, the agent can perform efficient end-to-end reasoning.

[0051] reinforcement learning reward mechanism design, defining a variety of situations that may occur during the execution of the measurement task by the agent and designing corresponding reward signals r.

[0052] hybrid action agent training based on parameterized Q-Learning, including: obtaining the state observation at the current time from the simulator and inputting it to the agent, the agent outputting the probe action , the simulator executing and obtaining the next state and the corresponding reward , while outputting an indication of whether the interaction is over , calculating the expected return estimate of the action that can obtain the maximum cumulative reward from the current time to the time when the interaction terminates , calculating the loss function according to , and updating the position predictor and the action selector according to the loss function. The depth reinforcement learning reward signal is designed in combination with the actual measurement target, and the agent trained by the reward signal can balance measurement efficiency and safety during measurement path planning.

[0053] Based on the characteristics of the measurement task, a reinforcement learning agent (Agent) based on the Transformer neural network architecture is constructed, a reward (Reward) signal is designed in combination with the actual measurement task target, and the agent is trained by reinforcement learning, ultimately realizing adaptive workpiece recognition, measurement path planning, and probe obstacle avoidance functions of the three-coordinate measuring machine during the measurement process of complex workpieces, and is suitable for scenes such as mechanical manufacturing, precision machining, quality detection, and other scenes that require high-precision geometric dimension measurement.

[0054] The online measurement step comprises:

[0055] (21) obtaining the to-be-measured feature geometry information from the workpiece design drawing, obtaining the probe information and the safety plane information from the three-coordinate measurement, and constructing a state vector .

[0056] (22) inputting the state vector into a position predictor, and the position predictor outputting a moving position in sequence according to the state vector , an action selector finding all possible actions of the probe moving from the current state to the moving position, calculating an expected return estimate of each action, and finding the action with the maximum expected return estimate . .

[0057] (23) controlling the probe to perform the action , and deleting the to-be-measured feature geometry information corresponding to the action.

[0058] (24) updating the state vector according to the execution result of step (23), and repeating steps (22)-(23) until a termination condition is met to terminate the execution.

[0059] In some embodiments, the state vector is constructed in the step of constructing the state vector .

[0060] The workpiece design drawing can be, but is not limited to, a CAD drawing.

[0061] In some embodiments, the to-be-measured feature geometry information, the probe information (position, direction, and optional rotation angle), and the safety plane information in the workpiece CAD are read as state information describing the current measurement machine, the information is characterized into a dimensionally aligned state vector using a corresponding embedding model, and the state vector is used as the input of the subsequent agent. Specifically, let the set of to-be-measured features be , where n is the number of to-be-measured features, which is different for different workpieces, and each has different dimensions according to different geometric features (for example, a cylinder can be represented by a center point, a depth, a radius, and an axial direction, while a sphere can be represented by a center point and a radius); the set of optional angles of the probe is , where is the direction vector of the optional angle; the probe information is , including the position, direction, and length of the current probe; the safety plane information is , representing the projection distance of the current probe from the workpiece to other safety planes except . Then the original state information can be represented as:

[0062] ;

[0063] in for The number of features.

[0064] Constructing state vectors The steps also include building an embedded model library. The acquired raw state information is mapped into dimension-aligned state vectors, which are the state vectors. .

[0065] .

[0066] Where d is the fixed state dimension, and N is the number of vectors that make up the state. The value of N is different for different states.

[0067] This embodiment designs the action space of the intelligent agent based on the movement mode of the probe of a coordinate measuring machine, defined as A. This includes the first-level action space. .

[0068] The parameters in the primary motion space represent the probe's movement, rotation, and measurement, respectively. The secondary motion space corresponding to each primary motion is... ,in:

[0069] when hour, The continuous motion space represents the three-dimensional coordinates of the probe movement.

[0070] when hour, For discrete motion space, it means selecting a predefined probe angle and rotating the probe to that angle.

[0071] when hour, The discrete motion space represents selecting a geometric feature on a workpiece to be measured and then measuring that feature. The motion space is defined as follows: Figure 1 As shown.

[0072] The state space S has a varying number of state features, therefore the state vector S can be considered a variable-length sequence. The action space A is a hybrid action space, and since the workpiece's selectable geometric features and predefined probe angles may both have differences in the number of states, A can also be considered a variable-length hybrid action space. To address the variable-length problem, this solution designs and constructs an end-to-end intelligent agent neural network architecture based on the sequence-independent property of Transformer. This architecture mainly consists of two parts: a position predictor Pos(·) and an action selector Act(·). The position predictor, based on the state vector... Predict the position of the probe movement.

[0073] In some embodiments, in constructing the end-to-end agent neural network architecture, the position predictor Pos(·) is used to predict the moving position of the probe according to the state vector , and the moving position prediction value .

[0074] , wherein .

[0075] The action selector Act(·) is used to find all possible actions of the probe from the agent action space A according to the state vector and the moving position prediction value , and calculate the expected return estimate of each action.

[0076] .

[0077] In order to make the agent complete the measurement of all features in the workpiece with the smallest movement cost under the premise of no collision, the present application summarizes six situations that may occur during the measurement task of the agent according to the actual measurement process and designs corresponding reward signals.

[0078] During the training process, the state observation at the current time is obtained from the simulator , the agent gives the probe action , the simulator executes and obtains the next state and the corresponding reward , and gives a representation d of whether the interaction is over. The five-tuple is recorded to the experience buffer every round of interaction, and multiple rounds of interaction are performed until the end. Figure 3 and Figure 4 show the curves of the average reward and the average task completion rate of the agent changing with the training round.

[0079] In some embodiments, in the step of designing the reinforcement learning reward mechanism, the reward signal r includes: completing the measurement of all features ; completing the measurement of a single feature ; the probe colliding ; moving ; rotating and illegal action , wherein each reward signal satisfies:

[0080] ;

[0081] is negatively correlated with the moving distance.

[0082] In some embodiments, in the hybrid action agent training step based on parameterized Q-Learning, the estimation of the maximum cumulative reward that the agent can obtain from time t to the end of the interaction T in the interaction at time t is: .

[0083] wherein, is the discount, is the reward at time t, d t is an indication of whether the interaction at time t is over, is the estimation of the cumulative discounted reward that can be obtained in the future for all possible actions a' at time t+1.

[0084] In some embodiments, in the hybrid action agent training step based on parameterized Q-Learning, the loss function is calculated as:

[0085] .

[0086] wherein θ is the policy network parameter, is the cumulative discounted reward estimation function based on the current policy network parameter θ, and are the expectations of the value in the action space A and the experience buffer D, respectively.

[0087] In some embodiments, in step (23), if the action of a is moving or rotating, the stylus information and the corresponding safety plane information also change, and the stylus information and the safety plane information are updated.

[0088] After training, the reasoning process of the agent is as follows:

[0089] Obtain the geometric information of the feature to be measured from the workpiece design drawing, obtain the stylus information and the safety plane information from the three-coordinate measurement, and construct the state vector S.

[0090] Input S into the position predictor, and the position predictor outputs the moving position according to the state vector The action selector finds all possible actions of the stylus moving from the current state to the moving position, calculates the expected return estimate of each action, and finds the action with the maximum expected return estimate.

[0091] Control the stylus to perform the action a and delete the corresponding geometric information of the feature to be measured. If a is "measurement" and the corresponding feature has not been measured, mark the feature as completed and delete it from the feature to be measured; if it is "moving" or "rotating", update the stylus information and the safety plane information.

[0092] When the feature to be measured is empty (trigger​ ) or a collision (trigger ) occurs, the reasoning is terminated and automatic 3D measurement is realized.

[0093] Of course, the above description is not a limitation on the present application, and the present application is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the spirit and scope of the present application should also be within the protection scope of the present application.

Claims

1. An automatic coordinate measuring method based on deep reinforcement learning, characterized in that, The process includes a deep learning model training step and an online measurement step. The deep learning model training step includes: Constructing state vectors The state vector This includes the geometric information of the feature to be measured, the probe information, and the safety plane information; Design the action space A of the intelligent agent, including the first-level action space. and secondary action space The first-level action space contains several actions, and the second-level action space contains parameter information corresponding to each action. Constructing an end-to-end agent neural network architecture, including a position predictor and action selector The action and its corresponding parameter information are stored separately, and values ​​are retrieved from the two action spaces and combined when needed. The reinforcement learning reward mechanism is designed by defining the situations that various agents may encounter during the execution of measurement tasks and designing corresponding reward signals r. Training of a hybrid action agent based on parameterized Q-Learning includes: obtaining the current state vector from the simulator. The data is then input to the agent, which outputs the probe action. The simulator executes And obtain the state vector at the next time step. and corresponding rewards It also outputs an indication of whether the interaction has ended. Calculate the expected reward estimate for the action that yields the maximum cumulative reward from the current moment until the interaction ends. ,according to Calculate the loss function and update the location predictor and action selector based on the loss function; Online measurement steps include: (21) Obtain the geometric information of the feature to be measured from the workpiece design drawing, obtain the probe information and safety plane information from the coordinate measuring machine, and construct the state vector. ; (22) will The input position predictor outputs the movement position sequentially based on the state vector. The action selector identifies all possible actions the probe can take from its current state to its new position, calculates the expected reward estimate for each action, and finds the action with the highest expected reward estimate. ; (23) Control the probe to perform actions And delete the corresponding geometric information of the feature to be tested; (24) Update the state vector according to the execution result of step (23), and repeat steps (22)-(23) until the termination condition is met and the execution is terminated; The termination condition is: the geometric information of the feature to be tested is empty or a collision occurs.

2. The automatic coordinate measuring method according to claim 1, characterized in that, Constructing state vectors In this process, the geometric information of the feature to be measured, the probe information, and the safety plane information directly obtained from the workpiece design drawing constitute the original state information. ; Constructing state vectors The steps also include building an embedded model library. The acquired raw state information is mapped into dimension-aligned state vectors, which are the state vectors. ; ； Where d is the fixed state dimension, and N is the number of vectors that make up the state.

3. The automatic coordinate measuring method according to claim 1, characterized in that, In building an end-to-end agent neural network architecture, the position predictor Used based on the state vector Predict the movement position of the probe, and give the predicted movement position value. ; ； in, ; Action Selector Used according to the state vector and predicted location Find all possible actions of the probe in the agent's action space A, and calculate the expected reward estimate for each action: 。 4. The automatic coordinate measuring method according to claim 1, characterized in that, In the design steps of a reinforcement learning reward mechanism, the reward signal r includes: completing the measurement of all features. ; Complete the measurement of a single feature The probe collided. ;move Rotation and illegal actions Among them, each reward signal satisfies: ； It is negatively correlated with the distance traveled.

5. The automatic coordinate measuring method according to claim 1, characterized in that, In the training steps of a hybrid action agent based on parameterized Q-Learning, the maximum cumulative reward that the agent can obtain from time t to the end of time T during the interaction at time t is estimated as follows: ； in, For discounts, Let d be the reward at time t. t Let t represent whether the interaction has ended. This is an estimate of the cumulative discounted reward that can be obtained in the future for all possible actions a' at time t+1. Let be the state vector at time t+1. The position of the probe at time t+1.

6. The automatic coordinate measuring method according to claim 5, characterized in that, In the training steps of the hybrid action agent based on parameterized Q-Learning, the loss function is calculated as follows: ； Where θ represents the policy network parameters. Let θ be the cumulative discount reward estimation function based on the current policy network parameters. and In the action space A and the experience buffer D respectively, The expected value is calculated, and the experience buffer D is used to store the quintuples recorded during multiple rounds of interaction between the agent and the simulator environment. , where d indicates whether the interaction has ended.

7. The automatic coordinate measuring method according to claim 1, characterized in that, In step (23), if When the action is movement or rotation, it also includes updating probe information and safety plane information.

Images