Method and device for realizing accurate hoisting of steel plate by automatically adjusting magnetism

Abstract

The invention discloses a method and device for achieving accurate lifting of a steel plate through automatic magnetism adjustment, and belongs to the technical field of steel plate lifting control. The method comprises the steps that core parameters of an electromagnet of an electromagnet lifting appliance and observation data in the historical lifting process are collected; the observation data in the historical hoisting process are preprocessed; constructing a reinforcement learning model, and training the constructed reinforcement learning model by using the preprocessed observation data in the historical hoisting process; wherein the reinforcement learning model adopts a self-adaptive learning rate and updating frequency; and the specification and number of the current to-be-hoisted steel plates are obtained, the optimal current of the electromagnet is obtained through the trained reinforcement learning model, and steel plate hoisting is achieved. According to the technical scheme, the electromagnetic attraction can be intelligently and accurately controlled, and the hoisting efficiency and precision are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of steel plate hoisting control, and in particular to a method and a device for realizing accurate hoisting of steel plates by electric automatic magnetic adjustment. Background Art

[0002] At present, most steel mills use electromagnetic hoists for steel plate adsorption and lifting. Electromagnetic hoists are lifting equipment that uses electromagnets to generate magnetic force to lift objects. In actual lifting, it is relatively easy to use electromagnetic hoists to lift a single steel plate placed separately, but it is difficult to accurately lift one or more steel plates stacked together as required. This is because, when accurately lifting one or more steel plates from a stack of stacked steel plates, it is usually necessary to rely on manual experience to adjust the current to change the suction force of the electromagnet. The current required to lift the steel plate is related to the type, specification, quantity, weight, etc. of the steel plate, and is not a simple linear relationship. When manually adjusting the magnetic lifting based on manual experience, the adjusted magnetic force is often too large, resulting in the lifting of excess steel plates, or the magnetic force is insufficient to lift the steel plates, and manual repeated adjustments are required, resulting in low lifting efficiency and delays in on-site logistics transportation. Summary of the invention

[0003] The present invention provides a method and device for realizing precise lifting of steel plates by automatic magnetic adjustment, so as to solve the technical problem that in the current electromagnet lifting process, manual magnetic adjustment and lifting are carried out manually relying on experience, and it is difficult to quickly and accurately control the lifting magnetic force of the electromagnetic crane according to the lifting requirements. The present invention realizes the technical effect of intelligent and precise control of the electromagnetic suction force of the electromagnetic crane and improving the lifting efficiency and accuracy.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] On the one hand, the present invention provides a method for automatically adjusting magnetic field to realize accurate lifting of steel plates, comprising:

[0006] Collect the core parameters of the electromagnetic hoist and the observation data during the historical lifting process;

[0007] Based on the core parameters of the electromagnets of the electromagnetic hoist, the observation data of the historical lifting process is preprocessed;

[0008] A reinforcement learning model is constructed and trained using the preprocessed observation data from the historical lifting process. The intelligent agent is trained by continuously interacting with the environment and an adaptive learning rate and update frequency are designed to promote the intelligent agent to effectively learn a continuous current regulation strategy. After the current is adjusted by the current regulation strategy, the magnetic force generated by the electromagnet meets the requirements of lifting steel plates of different thicknesses and quantities.

[0009] Obtain the specifications and quantity of the steel plates to be hoisted, and use the trained reinforcement learning model to obtain the optimal current of the electromagnet based on the specifications and quantity of the steel plates to be hoisted, so as to realize the lifting of the steel plates.

[0010] Furthermore, the core parameters of the electromagnet of the electromagnet hanger include: number of coil turns, working current, working air gap length, air air gap length, electromagnet core magnetic permeability and air magnetic permeability.

[0011] Furthermore, the observation data includes: specifications and quantity of the hoisted steel plates, and required current and magnetic data; wherein the steel plate specifications include: length, width, thickness, density and magnetic permeability of each layer of steel plate.

[0012] Furthermore, based on the core parameters of the electromagnet of the electromagnetic hoist, the observation data in the historical lifting process is preprocessed, including:

[0013] The missing values ​​in the observation data are processed by interpolation, mean filling or direct deletion;

[0014] By setting upper and lower limits of data, outliers in the observed data can be eliminated;

[0015] The Min-Max algorithm is used to normalize the observed data;

[0016] The observed data are smoothed using a Kalman filter recursive algorithm, wherein the Kalman filter recursive algorithm uses the collected current and magnetic data as observation quantities, and uses the functional relationship between magnetic force and current as a state prediction equation to predict magnetic force and current based on the core parameters of the electromagnet of the electromagnet hanger.

[0017] Furthermore, the functional relationship between the magnetic force and the current is expressed as:

[0018]

[0019] Among them, M k is the magnetic force required for lifting; n is the total number of steel plates to be lifted; N is the number of coil turns; I is the working current; μ0 is the air permeability; μ1 is the magnetic permeability of the electromagnet core; l1 is the working air gap length; l2 is the air gap length; V i is the volume of the i-th layer of steel plate, V i = l i w i t i , l i is the length of the i-th layer of steel plate, w i is the width of the i-th layer of steel plate, t i is the thickness of the i-th steel plate; e is the base of a natural number; is the attenuation coefficient, which indicates the attenuation degree of magnetic induction intensity when the electromagnet passes through each layer of steel plate, μ i is the magnetic permeability of the i-th steel plate; It represents the volume of the i-th layer of steel plate. The magnetic field corresponding to the volume has directionality, emphasizing the contribution of the directionality of the magnetic field at each point in this regional space.

[0020] Furthermore, the reinforcement learning model uses the thickness of the steel plate, the number of steel plates, the current of the electromagnet and the vector composed of the magnetic force of the electromagnet as the state space; and uses the adjustment amount of the current of the electromagnet as the action space.

[0021] Furthermore, the reward function of the reinforcement learning model is expressed as:

[0022]

[0023] Among them, R t is the reward function value; α is the penalty coefficient; F t is the current magnetic force; F g is the expected target magnetic force; δ is the coefficient controlling the response time reward; T g is the time step when the target magnetic force is reached; T0 is the time step when the current is adjusted; Indicates the speed of reaching the target magnetism. The faster the target magnetism is reached, the higher the reward.

[0024] Furthermore, during the training process, the reinforcement learning model uses the gradient descent method to minimize the loss function of the Critic network and update the Critic network; uses the policy gradient method to update the parameters of the Actor network; and updates the target network parameters through soft updating so that the target network parameters follow the parameters of the main network.

[0025] Furthermore, the adaptive learning rate and update frequency include:

[0026] Initialize learning rate and adjust: Set the initial learning rate of the Critic network to a value higher than the first preset threshold, and the initial learning rate of the Actor network to a value lower than the second preset threshold;

[0027] Adjust the learning rate according to different states: when the difference in thickness of the steel plates is greater than the third preset threshold and the number is greater than the fourth preset threshold, increase the learning rate of the Critic network; when the number of steel plates is not greater than the third preset threshold or the state change is less than the fifth preset threshold, reduce the learning rate of the Critic network;

[0028] The update frequency is adjusted by monitoring the control deviation or adaptively, including: 1) monitoring the learning process to adjust the update frequency: when the number of steel plate configuration states reaches a certain value, monitor the error change of the Q value; adjust the update frequency of the Actor network according to the current control error, that is, the deviation between the target magnetic force and the actual magnetic force; when the deviation between the magnetic force generated by the electromagnet and the target magnetic force is greater than the sixth preset threshold, shorten the update interval of the Actor network; when the magnetic force control of the electromagnet is close to the target, extend the update frequency of the Actor network; 2) adaptively adjust the update frequency: adjust the update frequency of the Actor network according to the stability of the Critic network; if the Q value of the Critic network is maintained within the preset range, increase the update frequency of the Actor network, otherwise, delay the update; the delayed update strategy is: set the parameter K, indicating that the Actor network is updated once after the Critic network is updated K times, and the update frequency of the Actor network is adaptively adjusted according to the complexity of the system, that is, when the state change is greater than the seventh preset threshold, increase the update frequency of the Actor network; when the state tends to be stable, reduce the update frequency.

[0029] On the other hand, the present invention also provides a device for implementing the above method to automatically adjust the magnetic field and realize precise lifting of steel plates, the device comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0030] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0031] The present invention combines the deep deterministic policy gradient DDPG algorithm with a neural network, designs an adaptively optimized learning rate and update frequency, improves the convergence speed and implementation accuracy of the reinforcement learning algorithm, and realizes the control of the complex relationship between steel plates of different specifications and quantities and the electromagnet current and magnetic force. The trained policy network is integrated into the PLC system. Through the real-time control of the PLC system, the automatic adjustment of the magnetic force is realized, thereby supporting the electromagnetic hoist to accurately lift multiple steel plates, greatly improving the lifting accuracy and efficiency, avoiding failures such as steel plate adhesion during lifting, and thus effectively reducing operating costs and improving economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1It is a flow chart of a method for realizing accurate lifting of steel plates by automatic magnetic adjustment provided by an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of the AO-DDPG algorithm provided by an embodiment of the present invention;

[0035] Figure 3 It is a structural block diagram of a device for automatically adjusting magnetic field to realize precise lifting of steel plates provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0038] First embodiment

[0039] This embodiment provides a method for automatic magnetic adjustment to achieve precise lifting of steel plates, so as to intelligently and dynamically adjust the electromagnetic force as needed to achieve fast and precise lifting of steel plates.

[0040] The method can be implemented by an electronic device, and the execution process of the method is as follows: Figure 1 As shown, the following steps are included:

[0041] S1, collects the core parameters of the electromagnetic hoist and the observation data during the historical lifting process;

[0042] Among them, the core parameters of the electromagnet of the electromagnet hoist include: the number of coil turns N; the working current I, unit A; the working air gap length l a , unit: mm; air gap length: l b , unit: mm; magnetic permeability of electromagnet core: μ e , air magnetic permeability μ0. The observation data include: the specifications and quantity of the hoisted steel plates, as well as the required current and magnetic force data; among which, the steel plate specifications include: the length l of the i-th layer of steel plate i , unit m; width w i , unit m; thickness t i , unit m; density ρ i , unit: kg / m 3 ; Magnetic permeability μ of the i-th steel plate i .

[0043] S2, based on the core parameters of the electromagnet, preprocesses the observation data during the historical lifting process;

[0044] Among them, the observation data in the historical lifting process is preprocessed, including:

[0045] The unreasonable data in the observation data are processed by interpolation, mean filling or direct deletion. By setting reasonable upper and lower limits, the extreme values ​​such as abnormal thickness or quantity of steel plates in the observation data are eliminated. Min-Max is used to normalize the thickness, quantity, current and magnetic quantity of steel plates so that all input feature scales are the same, which is convenient for network processing. The normalization formula is as follows:

[0046]

[0047] The Kalman filter recursive algorithm is used to smooth the data, so that the input data of the reinforcement learning is more accurate; wherein, the Kalman filter recursive algorithm uses the collected current and magnetic data as the observed quantity; and uses the functional relationship between the magnetic force and the current as the state prediction equation, and predicts the magnetic force and the current according to the core parameters of the electromagnetic hoist electromagnet; the specific processing method includes:

[0048] 1) Define the state vector and observation.

[0049] Define the state vector I k Indicates current, M k Represents magnetic force.

[0050] Defining Observables I m and M m Represents the collected current and magnetic data.

[0051] 2) Establish magnetic force function and current function as state prediction equations to predict magnetic force and current.

[0052] According to the safety factor requirements of the crane electromagnet, the magnetic force for lifting the steel plate is set to 3 times the weight of the steel plate. Assuming that the total number of layers of steel plates lifted by the electromagnet sling is n, the magnetic force function is obtained according to the principle of force balance:

[0053]

[0054] Where g is the acceleration due to gravity; V i represents the volume of the i-th steel plate; ρ i Represents the density of the i-th layer of steel plate.

[0055] The initial magnetic induction intensity B0 generated by the electromagnet is:

[0056]

[0057] The attenuation coefficient α of the magnetic induction intensity B when the electromagnet passes through each layer of steel plate i , i∈(1,n), attenuation coefficient α i It mainly depends on the magnetic permeability and thickness of the steel plate. Assume that the initial magnetic induction intensity is B0, and the magnetic induction intensity before the magnetic field passes through the i-th steel plate is B i , the magnetic induction intensity after the magnetic field passes through the i-th steel plate is B i+1 Since the magnetic field passes through each layer of steel plate, the magnetic induction intensity decreases according to the law of exponential decay, so the magnetic induction intensity after the magnetic field passes through the i-th layer of steel plate can be expressed as Then the final magnetic induction intensity B of the magnetic field passing through the nth steel plate n for:

[0058]

[0059] Assume that the magnetic force on the i-th steel plate is F i , then F i It can be calculated by the following formula:

[0060]

[0061] V i = l i w i t i ;

[0062] Calculate the magnetic force on all steel plates when the electromagnet lifts n layers of steel plates

[0063] Based on the above, combined with the magnetic model, the above force balance principle formula and the safety factor, the relationship expression between the current, the suction force and the number of steel plates is established:

[0064]

[0065] Among them, M k is the magnetic force required for lifting; n is the total number of layers (quantity) of the lifted steel plates; N is the number of coil turns; I is the working current; μ0 is the air permeability; μ1 is the magnetic permeability of the electromagnet core; l1 is the working air gap length; l2 is the air gap length; V i is the volume of the i-th layer of steel plate, V i = l i w i t i , l i is the length of the i-th layer of steel plate, w i is the width of the i-th layer of steel plate, ti is the thickness of the i-th steel plate; e is the base of a natural number; is the attenuation coefficient, which indicates the attenuation degree of magnetic induction intensity when the electromagnet passes through each layer of steel plate, μ i is the magnetic permeability of the i-th steel plate; It represents the volume of the i-th layer of steel plate. The magnetic field corresponding to the volume has directionality, emphasizing the contribution of the directionality of the magnetic field at each point in this regional space.

[0066] 3) Establish the covariance prediction equation.

[0067]

[0068] Among them, P k\k-1 It represents the covariance matrix corresponding to the prediction of the previous plate configuration state k-1 under the plate configuration state, which illustrates the uncertainty of the estimation of the plate under the k configuration state, and the prediction is based on no actual measurement value. k-1 Indicates the updated covariance matrix at the steel plate configuration state k-1, which means the uncertainty of the state estimation after combining the actual measurement value. k represents the state transfer matrix, It means that the covariance matrix of the previous configuration is passed through F k Transform to get the predicted covariance. Q is the transpose of the state transfer matrix, which is used to transform the transformed covariance matrix back to the state space, reflecting how the uncertainty of the state estimate propagates over time. k represents the covariance matrix of the process noise, represents the random error in the state transition process, Q k Through variance estimation, it is shown as follows:

[0069]

[0070] Among them, w i is the mean prediction error, and N is the number of data points.

[0071] 4) Calculate the Kalman gain and update the state estimate.

[0072] First, the Kalman gain is calculated using the following formula:

[0073]

[0074] Among them, K k represents the Kalman gain, which is used to balance the weights of predicted values ​​and observed values; represents the transpose of the observation matrix, R k Represents the observation noise covariance matrix, -1 means performing an inverse operation on the matrix;

[0075] Obtain the weights needed to update the state estimate to ensure that uncertainty is properly reflected when fusing the prediction and observation data; then update the state estimate using the following formula:

[0076]

[0077] in, is predicted as a state vector, which represents the predicted values ​​of current I and magnetic force M when the steel plate is in configuration state k. It is the calculation result combining the actual observed value and the predicted value. is the predicted value calculated according to the state prediction equation when the steel plate is in configuration state k. k It is the actual collected current and magnetic force data. k Represents the state matrix, which is used to transform the state vector Converted into observation space, reflecting the measured relationship between current and magnetic force.

[0078] 5) Update the covariance matrix.

[0079] In the update step, by combining the predicted values ​​with the actual observations, a more accurate covariance matrix is ​​obtained to help adjust the state estimate in order to continuously optimize the control decision.

[0080] P k =(IK k H k ) k|k-1 ;

[0081] 6) Implement Kalman filtering in a loop.

[0082] According to the configuration status of different numbers and thicknesses of steel plates, the magnetic and current functions are used to update the predicted values, and the status is updated in combination with the actual observed values ​​to complete the smoothing of all historical observation data.

[0083] S3, build a reinforcement learning model, use the pre-processed observation data of the historical lifting process to train the constructed reinforcement learning model, through the continuous interaction between the intelligent agent and the environment, and design an adaptive learning rate and update frequency, to promote the intelligent agent to effectively learn a continuous current regulation strategy, so that after the current is adjusted by the current regulation strategy, the magnetic force generated by the electromagnet meets the needs of lifting steel plates of different thicknesses and quantities;

[0084] Among them, the reinforcement learning model used in this embodiment is as follows Figure 2 The Adaptive Optimized DDPG algorithm (AO-DDPG) shown in the figure has the following construction process: construct a mathematical model, define the state space, action space and reward function, use neural networks to design state transfer functions, action strategies and evaluation strategies, and introduce the target network. The specific process includes:

[0085] 1) Define the state space: Assume that the thickness and quantity of the steel plate, the current and magnetic force of the electromagnet are continuous variables, and express the state as a vector:

[0086] St=[Q t ,H t ,I t ,F t ]

[0087] Q t : The current thickness of the steel plate

[0088] H t : The current number of steel plates

[0089] I t : Current current of the electromagnet

[0090] F t : The magnetic force generated by the current electromagnet

[0091] 2) Define the action space:

[0092] In order to ensure continuous regulation, the action is set to the change in current △I, that is, A t =△I t ;

[0093] A t Indicates the action of regulating the electromagnet current;

[0094] 3) Design reward function:

[0095] The design of the reward function takes the magnetic force reaching the expected target magnetic force as the core standard, while taking into account the time of the current regulation response. That is, if the target magnetic force is reached in a shorter time, the reward will be greater. The reward function is expressed as:

[0096]

[0097] Among them, R t is the reward function value; α is the penalty coefficient, the value range is [0,10], reflecting the significant impact of magnetic difference on the reward, adjusting the impact of this gap on the total reward, α value; F t is the current magnetic force; F g is the expected target magnetic force; δ is the coefficient for controlling the response time reward, with a value range of [0,1], which shows that the response time has a relatively small impact on the reward; T g is the time step when the target magnetic force is reached; T0 is the time step when the current is adjusted; Indicates the speed of reaching the target magnetism. The faster the target magnetism is reached, the higher the reward.

[0098] 4) Use neural networks to design state transfer functions, action strategies, and evaluation strategies, and introduce target networks:

[0099] A neural network is used as a state transfer function to fit the relationship between magnetic force, current, thickness and quantity of steel plates, that is, how to predict the magnetic force required for the next configuration state of the steel plate based on the current state and action.

[0100] The state transfer function is designed as:

[0101]

[0102] F t+1 : Represents the magnetic force in the configuration state of the steel plate at t+1, that is, the predicted output.

[0103] Represents a neural network function that accepts input S t , A t and parameter θ, which is used to fit the complex nonlinear relationship between magnetic force and current, steel plate thickness, and quantity.

[0104] S t : Represents the current state vector, which includes the main factors affecting the magnetic force, such as the number and thickness of steel plates, and represents a set of state descriptions.

[0105] A t : The action taken at time t, i.e. the current regulation △I t .

[0106] θ is the parameter of the neural network, obtained through training, aiming to optimize the neural network function To accommodate non-linear relationships in the training data.

[0107] Design action strategy, that is, Actor network design, to learn a strategy function through neural network, that is, to output a deterministic current adjustment value △I under the state St of given input steel plate quantity and thickness information t . A t =π(St;θ π ).

[0108] Design evaluation strategy, namely Critic network design, using neural network to estimate the state-action value function Q(S t ,A t ), evaluate the quality of the Actor network action, that is, evaluate whether the current regulation generated by the Actor can enable the electromagnetic hoist to generate enough magnetic force to lift the steel plate in time, so as to help the Actor network update its strategy and select better actions in the future.

[0109] Introduce the target network. In order to stabilize the training process, introduce the target Actor and target Critic network, and perform soft updates through the following formula to maintain stability.

[0110]

[0111] Q target represents the parameters of the target network, θ main Represents the parameters of the main grid, including the Actor network and the Critic network.

[0112] T represents the soft update coefficient, which controls the speed at which the target network follows the main network. The smaller the value, the faster and more stable the target network updates.

[0113] 1-T represents the weight of the target network in the previous steel plate configuration state.

[0114] By training the control strategy with the AO-DDPG algorithm, the control of the complex relationship between steel plates of different specifications and quantities and the electromagnet current and magnetic force can be optimized, thus realizing the automation of magnetic force regulation.

[0115] Specifically, the operation process of AO-DDPG algorithm training control strategy is as follows:

[0116] 1) Initialize the Actor network, Critic network, and target network.

[0117] Actor network initialization: Use a neural network, input the current state St, and output the current adjustment value △I t ;

[0118] Critic network initialization: input current state St and action A t , output the corresponding Q value, that is, evaluate the value of the state-action pair;

[0119] Target network initialization: Initialize the target network π that is the same as the Actor and Critic networks target and Q target , used to stabilize the training process.

[0120] 2) Establish an experience replay mechanism.

[0121] Create a replay buffer pool D to store states, actions, rewards, and next states (S t ,A t ,R t ,S t+1 ); Randomly sample a small batch of experiences from the replay buffer pool for training to avoid training instability caused by continuous data correlation.

[0122] 3) Update the Critic network:

[0123] Sample a batch of experience from the buffer, calculate the target value Q, and update the parameters of the Critic network by minimizing the loss function. The implementation is as follows:

[0124] ① Calculate the target Q value, the function expression is as follows:

[0125]

[0126] Among them, y t It is the desired target Q value, which is used to guide the Critic network to update parameters and make it as close to the target value as possible by minimizing the loss function.

[0127] R t Indicates current regulation A under the steel plate t configuration state t The direct return generated is calculated using the above reward function.

[0128] is the discount factor, which indicates the weight of future rewards, The larger the value, the more emphasis is placed on long-term future rewards, which means that the system will be more inclined to smooth and continuous control adjustments, maintain stable current adjustments, and avoid frequent and large current fluctuations to maintain the stability of the overall system; The smaller the value, the more the system pays attention to the accuracy and response of the instant current regulation. The current may be adjusted frequently and quickly to adapt to different steel plate specifications and changes to ensure the accuracy of each lifting.

[0129] ② Use the gradient descent method to minimize the loss function of the Critic network and update the Critic network to make the Q value estimation more accurate and improve the strategy performance.

[0130] The loss function L(θ Q ) is defined as the target Q value y t and the value Q(S) estimated by the current Critic network t ,A t θ Q ), the expression is:

[0131]

[0132] θ Q is the parameter of the Critic network, and the parameters are updated by gradient descent to minimize the loss function L(θ Q ), thereby improving the predictive ability of Q value;

[0133] i is the i-th sample in the batch, and each sample consists of state, action, reward and next state, that is,

[0134] is the target Q value of the i-th sample, which is calculated by the Critic network using the target network.

[0135] Represents the average loss, which averages the squared errors of all N samples to reduce the impact of fluctuations in parameter updates caused by a single sample and promote a more stable training process.

[0136] The parameter update formula of the Critic network is:

[0137]

[0138] ɑ Q It is the learning rate of the Critic network, which controls the magnitude of each update of the Critic network parameters.

[0139] Among them, the process of batch updating Actor network is as follows:

[0140] Update the parameters θ of the Actor network using the policy gradient method π , two gradients need to be calculated: 1) The gradient of the action is calculated through the Critic network. This gradient shows how the output current value affects the Q value under the current steel plate configuration state; 2) The gradient of the strategy is calculated through the Actor network. This gradient shows how the Actor network adjusts the parameter θ π 3) Combine the gradients of the Critic and Actor: Through the chain rule, the gradient of the Critic network to the action and the Actor network output action pair parameter θ π Gradient Combined, the update direction of the Actor network can maximize the Q value.

[0141] Parameter θ π The updated expression is as follows:

[0142]

[0143] Represents the policy gradient of the Actor network, representing the objective function J relative to the Actor network parameter θ π The goal is to maximize the expected reward J through the policy gradient, that is, by updating θ π Allow Actor networks to generate better current regulation.

[0144] J(θ π ) is the expected target, i.e., the objective function of the strategy, which represents the cumulative expected return on all steel plate configuration states and corresponding current regulation actions. It is calculated by the reward R of all future states.t Accumulated.

[0145] Represents the batch average of N samples. In reinforcement learning, experience back and batch updates are performed to break the time series correlation and provide training stability.

[0146] It represents the gradient of the Critic network to the current regulation, which measures the sensitivity (direction and magnitude of influence) of the current regulation A to the change of Q value (expected cumulative reward) under a given state S. The current regulation A is output by the current Actor network, that is, A = π(S; θ π ), Q is the evaluation result of the current regulation under the current state output by the Critic network.

[0147] represents the policy gradient of the Actor network, that is, the current regulation of the Actor network under a certain configuration state of the steel plate relative to the parameter θ π The derivative of , which reflects how the output of the Actor network changes with the network parameter θ π The gradient can be used to update the parameters of the Actor network so that the current regulation action output under the S strategy is adjusted in the direction of increasing the Q value (cumulative reward).

[0148] The Actor network parameter update formula is:

[0149]

[0150] ɑ π is the learning rate of the Actor network, which controls the amplitude of the Actor network parameter update.

[0151] 4) Furthermore, in order to improve the network learning effect, this embodiment designs an adaptive asymmetric update strategy to adjust the learning rate and adjustment frequency of the main grid to improve the convergence speed and control accuracy of the reinforcement learning algorithm, thereby promoting the intelligent agent to effectively learn a continuous current regulation strategy.

[0152] Among them, it should be noted that the existing asymmetric update strategies mainly use different update frequencies or rules, but most algorithms use the same learning rate, and after the update frequency, a fixed interval frequency (still a static frequency) is still used. In order to improve the network learning effect, the present invention closely combines the characteristics of multiple specifications and dynamic changes in steel plate lifting, and aims at the complex relationship between steel plate specifications, quantity and electromagnet current and magnetic force. Different learning rates and dynamically adjusted update frequencies are designed to improve the convergence speed and control accuracy of the reinforcement learning algorithm. The implementation method is as follows:

[0153] Initialize learning rate and adjust: Set the initial learning rate of the Critic network to a higher value ɑ Q , will Q When it is set to 0.005 and above, it can be regarded as a higher value, ɑ π The common value range of is [0.001, 0.01]; the initial learning rate of the Actor network is set to a lower value ɑ π , will π When it is set to 0.00005 or below, it can be regarded as a lower value. π The common value range of is [0.00001, 0.0001]. Adjust the learning rate according to different states: When the thickness difference of steel plates is large and the number is large, increase the learning rate of the Critic network to quickly adapt to the new state. Usually, when the thickness difference of steel plates exceeds 20-30mm, the system will become unstable or lack adaptability when responding quickly to steel plates of different thicknesses. The learning rate of the Critic network can be appropriately increased to accelerate its adaptation and magnetic control of steel plates of different thicknesses. If the change in steel plate thickness is very significant and the difference reaches 40mm or more, it is necessary to accelerate learning; when the number of steel plate configurations exceeds 5-10 layers, the magnetic attenuation is aggravated and the complexity of magnetic control increases. It is necessary to improve the learning efficiency in a timely manner. When the number of steel plate configurations is small, appropriately reduce the learning rate of the Critic network to avoid excessive updates.

[0154] By monitoring the control deviation or adaptively adjusting the update frequency, including: 1) Monitoring the learning process to adjust the update frequency: When the number of steel plates reaches a certain value, monitor the error change of the Q value. The value used to determine the number of steel plates to be monitored is not fixed, and varies due to the thickness and magnetic permeability of the steel plate. For example, for a 40mm thick steel plate, the Q value usually needs to be monitored at the 4th layer of steel plate. At this time, the magnetic force decays sharply, causing the error to begin to increase. According to the current control error, that is, the deviation between the target magnetic force and the actual magnetic force, adjust the update frequency of the Actor network. When the magnetic force generated by the electromagnet deviates greatly from the target magnetic force, shorten the update interval of the Actor network and accelerate the strategy adjustment; when the magnetic force control of the electromagnet is close to the target, extend the update frequency of the Actor network to reduce unnecessary frequent adjustments. Set the threshold of the magnetic force deviation according to the specific lifting task target requirements (generally 10%-20% of the target magnetic force value), and if this threshold is exceeded, it is determined that the magnetic force deviation is large. 2) Adaptive adjustment of the update frequency: adjust the update frequency of the Actor network according to the stability of the Critic network. If the Q value of the Critic network is relatively stable, the update frequency of the Actor network is appropriately increased, otherwise the update is delayed. The stability of the Q value is determined by the fluctuation amplitude during the Q value update process. It can be determined by observing the fluctuation of the Q value in multiple iterations. If the fluctuation amplitude remains unchanged or is small, it means that the convergence of the Q value becomes smaller. If the Q value tends to be stable in multiple training steps and no longer fluctuates greatly, it can be considered that the Q value tends to be stable. The delayed update strategy is to set the parameter K, which is usually set to a higher value of 5 or 10 times, indicating that the Actor network is updated once after the Critic network is updated K times. A higher K value makes the Q value estimated by the Critic network more stable, thereby providing a more reliable strategy guidance for the Actor network. The frequency of the Actor network update is adaptively adjusted according to the complexity of the system, that is, when the state changes greatly, such as the thickness and quantity of the steel plate change greatly, causing large changes in current, magnetic force, etc., the update frequency of the Actor network is increased; when the state tends to be stable, the update frequency is reduced to reduce unnecessary adjustments.

[0155] 5) Soft update target network: The target network parameters follow the parameters of the main network through soft update:

[0156]

[0157] in, and are the parameters of the target Critic network and the target Actor network, θ Q and θ π are the parameters of the Critic main network and the Actor main network respectively, and T is a small soft update coefficient (for example, 0.001) used to smooth the update of the target network.

[0158] S4, obtain the specifications and quantity of the steel plates to be hoisted, and use the trained reinforcement learning model to obtain the optimal current of the electromagnet according to the specifications and quantity of the steel plates to be hoisted, so as to realize the lifting of the steel plates.

[0159] It should be noted that during the training process, the agent continuously interacts with the environment, stores experience and updates the network until the strategy converges to the optimal strategy. Through the above training process, the agent can effectively learn a continuous current regulation strategy so that the magnetic force generated by the electromagnet meets the needs of lifting steel plates of different thicknesses and quantities. Based on this, this embodiment introduces the Actor-Critic architecture, combines the actual design of adaptive learning rate and update frequency for steel plate lifting, promotes DDPG to find the best current regulation strategy in a complex state space, and improves the stability of training through the experience replay mechanism and soft update of the target network.

[0160] In summary, this embodiment provides a method for automatic magnetic adjustment to achieve precise lifting of steel plates. The method for automatic magnetic adjustment to achieve precise lifting of steel plates combines the deep deterministic policy gradient DDPG algorithm with a neural network, designs an adaptively optimized learning rate and update frequency, improves the convergence speed and implementation accuracy of the reinforcement learning algorithm, and realizes the control of the complex relationship between steel plates of different specifications and different quantities and the electromagnet current and magnetic force, supports the electromagnet lifter to accurately lift multiple steel plates, greatly improves the lifting accuracy and lifting efficiency, and avoids failures such as steel plate suction and lifting adhesion, thereby effectively reducing operating costs and improving economic benefits.

[0161] Second embodiment

[0162] This embodiment provides a device for automatic magnetic adjustment to achieve accurate lifting of steel plates. The system structure of the device is as follows: Figure 3 As shown, the device includes: a PLC control system 310, a communication module 320, a user interface (UI) 330, a parameter database module 340, a current regulation module 350, a current detection module 360, a safety monitoring and diagnosis maintenance module 370 and a power management module 380; wherein each module respectively transmits data with the PLC control system 310 through the communication module 320; the functions of each component are as follows:

[0163] The PLC control system 310, which consists of a PCL controller, a control algorithm module and a feedback control module, is used to receive data from the current detection module, implement the control algorithm according to the model trained by AO-DDPG, and send control instructions to the current regulation module.

[0164] The communication module 320 ensures that all parts of the system work together and is responsible for transmitting data and control signals within the system. It can be a separate hardware device or integrated in the PLC control system or other modules. It supports multiple communication protocols and interfaces, such as Modbus, Profibus, Ethernet / IP, Fieldbus, etc.

[0165] User interface (UI) 330, UI is the medium for operators to interact with the PLC control system and is responsible for sending instructions and monitoring system status.

[0166] The parameter database module 340 is used to store system parameters and exchange data with the PLC control system through the communication module.

[0167] The current regulating module 350 regulates the current according to the instruction of the PLC control system.

[0168] Current detection module 360: provides real-time data of current size, detects the current size in the system, sends the current data to the PLC control system and the safety monitoring and diagnosis maintenance module, and provides real-time current feedback.

[0169] The safety monitoring and diagnostic maintenance module 370 exchanges information with the PLC control system through the communication module, including monitoring the system operation status to ensure operation safety, including overload protection, emergency stop and other functions, sending safety alarms to the PLC, triggering the PLC to execute the safety protocol; obtaining operation data and logs from the PLC, providing diagnostic information, assisting fault analysis, and helping maintenance personnel identify and solve potential problems.

[0170] Power management module 380: provides power status information to ensure stable operation of the system, manages power distribution and backup power switching. The power status information can be fed back to the PLC and the safety monitoring and diagnosis maintenance module. The PLC can also control power-related equipment.

[0171] Among them, the control algorithm module of the PLC system is used to convert the mathematical model into a specific control algorithm to realize automatic adjustment of the current. It integrates the trained AO-DDPG model, reads the status in real time and calculates the current adjustment action through the Actor network to dynamically adjust the current of the electromagnet. The magnetic force of the electromagnet can meet the needs of lifting steel plates of different thicknesses and quantities, and realize accurate lifting of the specified number of steel plates.

[0172] In addition, it should be noted that for the sake of convenience, Figure 3 Only the main parts of the device are shown. Moreover, the device for realizing accurate lifting of steel plates by automatic magnetic adjustment in this embodiment corresponds to the method for realizing accurate lifting of steel plates by automatic magnetic adjustment in the first embodiment described above; therefore, it will not be described in detail here.

[0173] In summary, this embodiment provides a device for automatic magnetic adjustment to achieve precise lifting of steel plates. The device for intelligent dynamic adjustment of electromagnetic force to achieve precise lifting of steel plates takes the PLC control system as the core, and converts the mathematical model of the above method into a specific control algorithm. It is integrated with the user interface (UI), current regulation module, current detection module, safety monitoring and diagnosis maintenance module, power management module, etc. to achieve automatic current adjustment and automatic change of electromagnet suction force, thereby achieving precise suction and lifting of a specified number of steel plates.

[0174] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.

[0175] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only an association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.

[0178] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0179] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0180] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0181] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0182] Finally, it should be noted that the above is only a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for ordinary technicians in this technical field, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A method for realizing accurate lifting of steel plates by automatic magnetic adjustment, characterized in that: include: Collect the core parameters of the electromagnetic hoist and the observation data during the historical lifting process; Based on the core parameters of the electromagnets of the electromagnetic hoist, the observation data of the historical lifting process is preprocessed; A reinforcement learning model is constructed and trained using the preprocessed observation data from the historical lifting process. The intelligent agent is trained by continuously interacting with the environment and an adaptive learning rate and update frequency are designed to promote the intelligent agent to effectively learn a continuous current regulation strategy. After the current is adjusted by the current regulation strategy, the magnetic force generated by the electromagnet meets the requirements of lifting steel plates of different thicknesses and quantities. Obtain the specifications and quantity of the steel plates to be hoisted, and use the trained reinforcement learning model to obtain the optimal current of the electromagnet based on the specifications and quantity of the steel plates to be hoisted, so as to realize the lifting of the steel plates.

2. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 1, characterized in that: The core parameters of the electromagnet of the electromagnet hoist include: number of coil turns, working current, working air gap length, air air gap length, electromagnet core magnetic permeability and air magnetic permeability.

3. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 1, characterized in that: The observation data include: the specifications and quantity of the hoisted steel plates and the required current and magnetic data; wherein the steel plate specifications include: the length, width, thickness, density and magnetic permeability of each layer of steel plate.

4. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 1, characterized in that: Based on the core parameters of the electromagnetic hoisting electromagnet, the observation data of the historical lifting process is preprocessed, including: The missing values ​​in the observation data are processed by interpolation, mean filling or direct deletion; By setting upper and lower limits of data, outliers in the observed data can be eliminated; The Min-Max algorithm is used to normalize the observed data; The observed data are smoothed using a Kalman filter recursive algorithm, wherein the Kalman filter recursive algorithm uses the collected current and magnetic data as observation quantities, and uses the functional relationship between magnetic force and current as a state prediction equation to predict magnetic force and current based on the core parameters of the electromagnet of the electromagnet hanger.

5. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 4, characterized in that: The functional relationship between the magnetic force and the current is expressed as: Among them, M k is the magnetic force required for lifting; n is the total number of steel plates to be lifted; N is the number of coil turns; I is the working current; μ0 is the air permeability; μ1 is the magnetic permeability of the electromagnet core; l1 is the working air gap length; l2 is the air gap length; V i is the volume of the i-th layer of steel plate, V i = l i w i t i , l i is the length of the i-th layer of steel plate, w i is the width of the i-th layer of steel plate, t i is the thickness of the i-th steel plate; e is the base of a natural number; is the attenuation coefficient, which indicates the attenuation degree of magnetic induction intensity when the electromagnet passes through each layer of steel plate, μ i is the magnetic permeability of the i-th steel plate; Represents the volume of the i-th layer of steel plate.

6. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 1, characterized in that: The reinforcement learning model uses the thickness of the steel plate, the number of steel plates, the current of the electromagnet and the vector composed of the magnetic force of the electromagnet as the state space; and uses the adjustment amount of the current of the electromagnet as the action space.

7. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 6, characterized in that: The reward function of the reinforcement learning model is expressed as: Among them, R t is the reward function value; α is the penalty coefficient; F t is the current magnetic force; F g is the expected target magnetic force; δ is the coefficient controlling the response time reward; T g is the time step when the target magnetic force is reached; T0 is the time step when the current is adjusted; Indicates the speed of reaching the target magnetism. The faster the target magnetism is reached, the higher the reward.

8. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 6, characterized in that: During the training process, the reinforcement learning model uses the gradient descent method to minimize the loss function of the Critic network and update the Critic network; uses the policy gradient method to update the parameters of the Actor network; and updates the target network parameters through soft updating so that the target network parameters follow the parameters of the main network.

9. The method for realizing accurate lifting of steel plates by automatic magnetic adjustment as claimed in claim 1, characterized in that: The adaptive learning rate and update frequency include: Initialize learning rate and adjust: Set the initial learning rate of the Critic network to a value higher than the first preset threshold, and the initial learning rate of the Actor network to a value lower than the second preset threshold; Adjust the learning rate according to different states: when the difference in thickness of the steel plates is greater than the third preset threshold and the number is greater than the fourth preset threshold, increase the learning rate of the Critic network; when the number of steel plates is not greater than the third preset threshold or the state change is less than the fifth preset threshold, reduce the learning rate of the Critic network; The update frequency is adjusted by monitoring the control deviation or adaptively, including: 1) monitoring the learning process to adjust the update frequency: when the number of steel plate configuration states reaches a certain value, monitor the error change of the Q value; adjust the update frequency of the Actor network according to the current control error, that is, the deviation between the target magnetic force and the actual magnetic force; when the deviation between the magnetic force generated by the electromagnet and the target magnetic force is greater than the sixth preset threshold, shorten the update interval of the Actor network; when the magnetic force control of the electromagnet is close to the target, extend the update frequency of the Actor network; 2) adaptively adjust the update frequency: adjust the update frequency of the Actor network according to the stability of the Critic network; if the Q value of the Critic network is maintained within the preset range, increase the update frequency of the Actor network, otherwise, delay the update; the delayed update strategy is: set the parameter K, indicating that the Actor network is updated once after the Critic network is updated K times, and the update frequency of the Actor network is adaptively adjusted according to the complexity of the system, that is, when the state change is greater than the seventh preset threshold, increase the update frequency of the Actor network; when the state tends to be stable, reduce the update frequency.

10. A device for automatic magnetic adjustment to achieve accurate lifting of steel plates, comprising a processor and a memory; wherein: At least one instruction is stored in the memory, wherein the instruction is loaded and executed by the processor to implement the method for automatic magnetic adjustment to achieve precise lifting of steel plates as described in any one of claims 1 to 9.

Images