Intelligent optimization method for superfine tungsten filament drawing process based on reinforcement learning

Abstract

The invention relates to the technical field of advanced manufacturing and artificial intelligence, and discloses a reinforcement learning-based superfine tungsten filament drawing process intelligent optimization method, which comprises the following steps of: constructing a digital twinborn model in a tungsten filament drawing process; based on the model, a hierarchical reinforcement learning framework is adopted for strategy training, and the framework comprises a high-level strategy network responsible for inter-pass macro target planning and a low-level strategy network responsible for real-time fine regulation and control in the pass to achieve the macro target. Before the low-layer strategy network outputs the control action, the disconnection risk of the action is predicted, and the high-risk action is corrected or denied, so that safe exploration and optimization are realized. Through combination of digital twinning, hierarchical decision and safety constraint, intelligent control over the whole flow of the superfine tungsten wire drawing process is achieved, and the yield, production efficiency and quality consistency of superfine tungsten wires of 30 microns and below can be remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of advanced manufacturing and artificial intelligence technology, and more specifically to an intelligent optimization method for ultrafine tungsten wire drawing process based on reinforcement learning. Background Technology

[0002] Against the backdrop of global energy structure transformation, the trend towards thinner silicon wafers in the photovoltaic industry has placed greater demands on the diameter of diamond wire, the cutting tool. Traditional high-carbon steel wire exhibits a sharp decline in performance once its diameter falls below 40 μm, while tungsten wire, due to its high melting point and high strength, has become an ideal alternative. However, tungsten has a high ductile-brittle transition temperature and exhibits significant brittleness at room temperature. Its ultra-fine wire drawing process has an extremely narrow window, involving complex thermo-mechanical coupling effects and microstructure evolution. Currently, it heavily relies on human experience, resulting in limited optimization dimensions, slow response, and difficulty in transferring experience. Existing automated control methods are mostly local response control, unable to achieve global multi-objective optimization. While reinforcement learning shows potential, the high risk involved in its exploration process hinders its practical application on expensive production lines.

[0003] Based on this, the present invention proposes an intelligent optimization method for ultrafine tungsten wire drawing process based on reinforcement learning to solve the above problems. Summary of the Invention

[0004] To overcome the aforementioned deficiencies in the prior art, this invention provides an intelligent optimization method for ultrafine tungsten wire drawing process based on reinforcement learning, in order to solve the problems existing in the background art.

[0005] This invention provides the following technical solution: an intelligent optimization method for ultrafine tungsten wire drawing process based on reinforcement learning, comprising the following steps: S1: Acquire real-time multimodal state data of the ultrafine tungsten wire drawing process; S2: Input the real-time multimodal state data into a pre-trained hierarchical reinforcement learning policy model, the model including a high-level policy network and a low-level policy network; S3: The high-level strategy network outputs macroscopic process objectives to guide the next pass or the remaining stages of the current pass based on the global state of the current drawing pass. S4: The low-level policy network generates candidate real-time control action parameters based on the real-time multimodal state data and the macroscopic process objective; S5: Input the candidate real-time control action parameters into a forward-looking safety assessment module to assess the probability of their corresponding disconnection risk; S6: If the probability of the disconnection risk is lower than the preset safety threshold, the candidate real-time control action parameter is used as the final action parameter; if it is higher, the candidate parameter is modified for safety and then used as the final action parameter. S7: Control the drawing equipment and online heat treatment equipment to perform operations according to the final action parameters.

[0006] As a further aspect of the present invention: the real-time multimodal state data includes: the tungsten wire diameter obtained by a laser diameter gauge; the drawing tension obtained by a tension sensor; the tungsten wire surface temperature obtained by an infrared thermometer; the acoustic emission signal characteristics characterizing the activity of microcracks inside the material obtained by an acoustic emission sensor; and the eddy current signal characteristics characterizing the conductivity and surface defects of the material obtained by an online eddy current flaw detection device.

[0007] As a further aspect of the present invention: the macroscopic process objective is a target vector, which includes a quantitative index of at least one material microstructure state expected to be achieved at the end of the next pass; the quantitative index is selected from: target average grain size, target dislocation density, target residual stress level, or target texture intensity.

[0008] As a further aspect of the present invention: the prospective security assessment module is an independent neural network model, which is obtained by supervised learning training on a production dataset containing historical disconnection events; the input of the security assessment module is a concatenated vector of the current state and candidate actions, and the output is the disconnection risk probability value in the interval [0,1].

[0009] As a further aspect of the present invention, the safety correction includes: when the probability of wire breakage is higher than the safety threshold, according to preset safety rules, forcibly reducing the pulling speed value and / or increasing the online heat treatment temperature value in the candidate real-time control action parameters until the probability of wire breakage is lower than the safety threshold after the corrected action parameters are evaluated.

[0010] A method for training an intelligent optimization method for ultrafine tungsten wire drawing process based on reinforcement learning includes the following steps: T1: Construct a digital twin model of the tungsten wire drawing process. The digital twin model can simulate and output the evolution of multimodal state data and the final product performance during the drawing process based on the input process parameters and the initial state of the material. T2: Define a hierarchical Markov decision process, where the reward function of the high-level strategy is calculated based on the tungsten wire quality, energy consumption, and processing time after completing the entire pass, and the reward function of the low-level strategy is calculated based on the extent to which its actions achieve the macroscopic process objectives set by the high-level strategy. T3: In the digital twin model environment, a course learning strategy is adopted, starting with simple pulling tasks with thicker wire diameter and lower speed, and gradually increasing the difficulty. Through interaction with the virtual environment, a reinforcement learning algorithm based on the actor-critic framework is used to iteratively train the high-level and low-level policy networks until the model converges.

[0011] As a further aspect of the present invention: the reward function of the lower-level strategy Defined as: ; in, The macroscopic process objective vector set by the high-level strategy in the k-th pass. For the state at the next moment The inferred material microstructure state vector, The KL divergence, which measures the difference between the two distributions, is used as the weighting coefficient for the degree of goal achievement. At the current drawing speed, Let represent instantaneous energy consumption, and β and γ be the corresponding weighting coefficients.

[0012] A smart optimization system for ultrafine tungsten wire drawing process based on reinforcement learning, comprising: A multimodal data acquisition unit is configured on the drawing production line to collect multimodal state data as described above in real time. The edge computing decision unit is communicatively connected to the data acquisition unit and has a pre-trained hierarchical reinforcement learning strategy model and a forward-looking security assessment module deployed within it, used to receive state data and generate final action parameters. The equipment control execution unit is used to receive the final action parameters and convert them into low-level control commands for the drawing equipment and the online heat treatment equipment.

[0013] As a further aspect of the present invention, it also includes a cloud-based training and digital twin platform, the platform comprising: A digital twin module is used to perform high-fidelity simulation of the drawing process according to the method described above; The model training module is used to pre-train, validate, and iteratively update the hierarchical reinforcement learning strategy model in the digital twin module. The model management and deployment module is used to distribute mature models trained in the cloud to the edge computing decision unit.

[0014] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the preceding descriptions.

[0015] The technical effects and advantages of this invention are as follows: By using hierarchical reinforcement learning to achieve multi-objective global optimization, the yield and production efficiency of ultrafine tungsten wires are significantly improved.

[0016] The introduction of a forward-looking safety assessment module ensures that the optimization process is carried out within the safety boundary, thus solving the problem of high exploration risk in reinforcement learning in industrial applications.

[0017] Utilize digital twin technology for low-cost, high-efficiency model training and iteration.

[0018] By using multimodal sensing and microstructure targets, the interpretability and controllability of the process are improved. Attached Figure Description

[0019] The invention will now be further described with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart of a method for intelligent optimization of ultrafine tungsten wire drawing process based on reinforcement learning, according to the present invention. Figure 2 This is a schematic diagram of the cloud-edge-device collaborative system architecture of the present invention; Figure 3 This is a flowchart of the forward-looking safety assessment and intervention mechanism of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. These embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0022] Example 1: Detailed Flow of the Intelligent Optimization Method This embodiment aims to illustrate in detail the execution flow of the intelligent optimization method described in this invention. (Refer to...) Figure 1 This method, driven by a pre-trained model deployed in an edge computing decision unit, executes the following steps in a loop: Step S1: Real-time multimodal state perception and data fusion High-precision sensor arrays are deployed at key stations in the multi-pass drawing production line for ultra-fine tungsten wires to construct a comprehensive state awareness network. In each decision-making and control cycle (e.g., every 100 milliseconds), the system performs data acquisition and fusion to form the current state vector St, as shown in the formula: ; in: Instantaneous diameter of the tungsten wire measured by an online laser diameter gauge; : Pulling force / tension measured by a piezoelectric / strain gauge sensor mounted on the guide wheel or drawing die base; Temperature distribution of tungsten wire at key points such as the wire drawing die exit, the heat treatment zone entrance, the middle of the heat treatment zone, and the heat treatment zone exit, as measured by a non-contact infrared thermometer array; Acoustic emission (AE) signals are collected by acoustic emission sensors attached to the wire drawing die holder or key guide wheel bearing seat, and feature vectors are extracted after signal processing. AE signals are extremely sensitive to dynamic processes such as microcrack initiation and dislocation movement inside the material. Feature vectors may include ring count, energy, amplitude distribution, etc., which can indirectly characterize the degree of damage accumulation in the material. The signal feature vector acquired by the eddy current (EC) probe surrounding the tungsten filament. The EC signal is sensitive to the electrical conductivity and surface / near-surface defects of the material. After the tungsten filament undergoes different degrees of work hardening and recrystallization, its electrical conductivity will change regularly. Therefore, the EC signal can be used to indirectly infer the microstructure of the material (such as dislocation density and degree of recrystallization). The ID or service life of the wire drawing die used in the current pass; The time or length of time that the current track has been completed.

[0023] Steps S2-S4: Hierarchical reinforcement learning decision-making will transfer the state vector The input is fed into a pre-trained hierarchical reinforcement learning model; this model consists of a high-level policy network. and low-level policy networks These components can all be implemented using deep neural networks, such as multilayer perceptrons (MLPs) or recurrent neural networks (RNNs) to process temporal information. High-level decision-making (Manager): At the beginning of each course k, the high-level policy network... It starts once; it receives the initial macroscopic state of that channel. ,For example: ; in, k and These are the starting and target line diameters for this course, respectively. This is the overall quality score of the tungsten wire produced in the previous run. It provides a summary of the previous processing history; the high-level strategy network outputs a macro-level process goal. : It is a vector that defines the expected internal state of the material at the end of this pass, for example: in, For the target average grain size, For the target dislocation density, The target residual stress level.

[0024] Low-level decision-making (Worker): During the execution of path k, the low-level policy network... A decision is made at each time step t. The input it receives is the current real-time state. Targets issued by higher management splicing: The low-level policy network outputs a candidate real-time control action vector. in, These are candidate pull-out speed settings. It is the candidate online heat treatment heating power setting value.

[0025] Steps S5-S6: Proactive safety assessment and intervention, such as Figure 3 As shown, candidate actions Instead of executing directly, it is combined with the current state. Together, they are fed into the forward-looking security assessment module Ψ. Ψ is an independent, fully trained binary or regression neural network. Its training data comes from historical production databases, particularly containing a large number of "state-action-result (whether disconnected)" samples. Through supervised learning, Ψ learns to predict the action to be performed in a specific state St. The probability of causing a disconnection : The system sets a safety threshold θsafe (e.g., 0.05). The action was deemed safe and adopted directly. Final Action . The action is deemed high-risk, and a safety intervention procedure is initiated. This procedure can be a simple set of rules or another optimization solver. The rules are: For example, iteratively reducing and / or improve Re-evaluate in each iteration until it is less than . in , It's a small adjustment step size.

[0026] Step S7: Control the execution of the finally determined safety actions. It is sent to the device control execution unit. This unit converts it into instructions that the PLC (Programmable Logic Controller) can understand, for example, to... Convert to the frequency setting value of the frequency converter. This is converted to the conduction angle setting value of the SCR (Silicon Controlled Rectifier) ​​power regulator, thereby precisely controlling the speed of the drawing motor and the current of the heating wire.

[0027] The entire process includes perception, decision-making, evaluation, and execution, ensuring that the pull-out process always operates on a dynamically optimized safety trajectory.

[0028] Example 2: Training Method of Hierarchical Reinforcement Learning Model and Digital Twin This embodiment details how to safely and efficiently train the hierarchical reinforcement learning model and security assessment module described in Embodiment 1 using digital twin technology.

[0029] T1: Construction of High-Fidelity Digital Twin Model A high-fidelity digital twin model synchronized with the physical production line is constructed; it serves as a virtual sandbox for training and optimization. This model is coupled with the following sub-models: Thermo-mechanical coupled finite element model: Based on commercial software (such as DEFORM, Abaqus) or self-developed code, an elastoplastic mechanical model of the drawing process is established. The model takes the drawing speed, die geometry parameters, and friction coefficient as input, and can output the drawing force, stress-strain field, and temperature rise caused by plastic deformation.

[0030] Heat transfer and heat treatment model: A comprehensive thermal model is established to calculate the heat generated by the deformation of the tungsten wire in the drawing die, the conduction / convection / radiation heat transfer with the die and the environment, and the energy injection of online induction heating, so as to accurately simulate the temperature field evolution of the tungsten wire throughout the entire process.

[0031] Microstructure evolution model: Based on physical metallurgy principles, this model integrates dynamic recovery, dynamic / static recrystallization kinetic models (such as the JMAK equation), and grain growth models. Using temperature and strain history as inputs, this model predicts the evolution of microscopic state variables such as average grain size and dislocation density.

[0032] Data-driven residual model: Using real production line data, a machine learning model (such as Gaussian process regression or neural network) is trained to learn and correct the systematic deviation between the above physical model and the measured data, ensuring the high fidelity of the digital twin output.

[0033] T2: Hierarchical Markov Decision Process and Reward Function Design. In a digital twin environment, the optimization problem is formalized as a hierarchical Markov decision process.

[0034] High-level MDP: state : Macroscopic state of the path.

[0035] action Macroscopic process objectives .

[0036] award : Distributed after stage k, based on the overall performance of that stage: ,in, The quality score of the produced tungsten wire can be calculated by simulation of microstructure uniformity, number of defects, etc. The total time taken for each round Total energy consumption, , , These are the weighting coefficients.

[0037] Low-level MDP: state : .

[0038] action Real-time control of actions .

[0039] award Employing a goal-oriented intrinsic reward mechanism encourages lower-level actions to achieve higher-level goals, while also considering efficiency and energy consumption. in, It is an auxiliary "state decoder" network that can determine the macroscopic state at the next time step. (Especially indirect signals such as acoustic emission and eddy current) can be used to infer the probability distribution of the current material microstructure state vector. It is the KL divergence, used to measure the relationship between the inferred state and the target. The distribution differences. α, β, γ are weighting coefficients.

[0040] T3: Curriculum-based reinforcement learning training is conducted on a cloud-based digital twin platform, employing reinforcement learning algorithms (such as SAC or TD3) suitable for continuous action spaces to train the HRL model. The training process uses a curriculum-based learning strategy, progressing from easy to difficult: Course Phase 1 (Basic): The simulation is scaled down from 200μm to 180μm, with a wider process window and low speed range set, allowing the agent to learn basic control and avoid severe disconnections.

[0041] Course Phase 2 (Advanced): Target wire diameter refinement, such as from 50μm to 40μm, increasing the speed limit and simulating more complex operating condition fluctuations.

[0042] Course Phase 3 (Expert): Challenge ultra-fine filament drawing, such as drawing from 35μm to 30μm, while maintaining an extremely high yield rate in high-efficiency production.

[0043] The security assessment module Ψ is trained synchronously during this stage, using massive amounts of "state-action-disconnection result" data generated by digital twins for supervised learning, enabling it to learn to accurately predict risks.

[0044] Example 3: Intelligent Optimization System Architecture This embodiment describes in detail the hardware and software system architecture for implementing the above method, referring to... Figure 2 The system adopts a collaborative design of "cloud-edge-device".

[0045] End-side (data acquisition and execution layer): Multimodal data acquisition unit (100): physically integrated into the drawing production line.

[0046] Laser diameter gauge (110): Real-time monitoring of tungsten wire diameter.

[0047] Tension sensor (120): High-frequency response sensor that captures dynamic changes in tension.

[0048] Infrared temperature measurement array (130): multi-point non-contact temperature measurement.

[0049] Acoustic emission detection unit (140): includes AE sensor, preamplifier and acquisition card.

[0050] Eddy current flaw detection unit (150): high-frequency eddy current probe and signal processing circuit.

[0051] Side (Intelligent Decision Layer): Edge computing decision unit (200): composed of high-performance industrial computers.

[0052] Data preprocessing and fusion module (210): Filters, extracts features and synchronizes the raw sensor data.

[0053] HRL Decision Engine (220): Load and run the trained high-level policy network π_H and low-level policy network π_L.

[0054] Security assessment engine (230): Loads and runs the security assessment module Ψ, performing risk assessment and action correction.

[0055] Real-time database (240): caches runtime data and supports online monitoring and short-term historical backtracking.

[0056] Cloud side (training and evolution layer): Cloud-based training and digital twin platform (400): Digital twin simulator (410): Runs a high-fidelity simulation model of the drawing process.

[0057] AI Model Trainer (420): Provides large-scale GPU computing power for offline training and hyperparameter optimization of HRL and security models.

[0058] Data Lake and Model Library (430): Archives all production data, simulation data and model versions.

[0059] Monitoring and Management Interface (440): Used for system status monitoring, model version management, digital twin parameter calibration and visualization of inference results.

[0060] Example 4: Closed-Loop Operation and Self-Evolution of the System This embodiment illustrates the complete closed-loop workflow and self-evolution mechanism of the system after deployment.

[0061] Online closed-loop optimization: Based on the process in Implementation Example 1, the system performs a millisecond-level "perception-decision-evaluation-execution" cycle at the edge, enabling real-time autonomous optimization of the production line.

[0062] The result after performing the action (new state) Data such as connection status and energy consumption are recorded and used to generate reward signals. This data can be periodically transmitted back to the cloud in batches.

[0063] Model self-evolution: Triggering conditions: The cloud platform continuously monitors the performance data transmitted back from the edge. When it detects a decrease in the model's average prediction reward, an increase in the false positive / false negative rate of the security assessment module, or a deviation between the prediction accuracy of the digital twin (such as force, temperature) and the real data exceeding a preset threshold, the model evolution process is automatically triggered.

[0064] Evolution process: Digital twin calibration: Using the latest production data, the physical parameters or residual model in the twin model are retrained to improve its fidelity.

[0065] Policy model fine-tuning: In the updated digital twin environment, the policy network of the HRL model is incrementally trained or fine-tuned using a large amount of recently collected interaction data to adapt it to new changes in equipment, raw materials or environment.

[0066] Security module update: The security assessment module Ψ is retrained using newly generated data, especially data containing new disconnection patterns, to enhance its risk identification capabilities.

[0067] Once the evolution is complete, the new model with verified performance improvements will be distributed to each decision-making unit on the edge through the model management and deployment module, completing the overall knowledge update and performance iteration of the system, thereby achieving continuous autonomous evolution.

[0068] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for intelligent optimization of ultra-fine tungsten wire drawing process based on reinforcement learning, characterized in that, The method comprises the following steps: S1: acquiring real-time multi-modal state data of the ultrafine tungsten wire drawing process; S2: inputting the real-time multi-modal state data into a pre-trained hierarchical reinforcement learning strategy model, the model comprising a high-level strategy network and a low-level strategy network; S3: the high-level strategy network outputs a macro process target for guiding the next pass or the remaining stage of the current pass based on the global state of the current drawing pass; S4: the low-level strategy network generates a candidate real-time control action parameter based on the real-time multi-modal state data and the macro process target; S5: inputting the candidate real-time control action parameter into a prospective safety assessment module to evaluate the corresponding risk probability of wire breakage; S6: if the risk probability of wire breakage is lower than a preset safety threshold, the candidate real-time control action parameter is taken as the final action parameter; if higher, the candidate parameter is safety corrected and taken as the final action parameter; S7: controlling the drawing equipment and online heat treatment equipment to perform operations according to the final action parameter.

2. The method of claim 1, wherein, The real-time multi-modal state data comprises: a tungsten wire diameter acquired by a laser diameter gauge; a drawing tension acquired by a tension sensor; a tungsten wire surface temperature acquired by an infrared temperature gauge; an acoustic emission signal feature representing the activity of internal micro-cracks of the material acquired by an acoustic emission sensor; and an eddy current signal feature representing the electrical conductivity and surface defects of the material acquired by an online eddy current flaw detection device.

3. The method of claim 1, wherein, The macro process target is a target vector comprising at least one quantitative index of a material microstructure state expected to be achieved at the end of the next pass; the quantitative index is selected from: a target average grain size, a target dislocation density, a target residual stress level, or a target texture intensity.

4. The method of claim 1, wherein, The prospective safety assessment module is an independent neural network model obtained by supervised learning training on a production data set containing historical wire breakage events; the input of the safety assessment module is a spliced vector of the current state and the candidate action, and the output is a wire breakage risk probability value in the interval [0, 1].

5. The method according to claim 1 or 4, characterized in that, The safety correction comprises: when the risk probability of wire breakage is higher than the safety threshold, the drawing speed value in the candidate real-time control action parameter is forcibly reduced and / or the online heat treatment temperature value is increased according to a preset safety rule until the wire breakage risk probability of the modified action parameter is lower than the safety threshold after evaluation.

6. A method for training the method of claim 1, characterized by, The method comprises the following steps: T1: constructing a digital twin model of the tungsten wire drawing process, the digital twin model being capable of simulating and outputting the evolution of multi-modal state data and the performance of the final product in the drawing process according to input process parameters and initial material state; T2: defining a hierarchical Markov decision process, wherein the reward function of the high-level strategy is calculated based on the tungsten wire quality, energy consumption and processing time after completing the entire pass, and the reward function of the low-level strategy is calculated based on the extent to which the action of the low-level strategy achieves the macro process target set by the high-level strategy. T3: In the digital twin model environment, a curriculum learning strategy is adopted, starting from a simple drawing task with a rough line diameter and low speed, gradually increasing the difficulty, interacting with the virtual environment, using reinforcement learning algorithms based on the Actor-Critic framework to iteratively train the high-level and low-level policy networks until the model converges.

7. The method of claim 6, wherein: The reward function of the low-level policy is defined as: ； wherein, is a macroscopic process target vector set by high-level policy at the kth pass, is a state of the next time instant is a material microstructure state vector inferred, is a KL divergence measuring the difference between the two distributions, and a is a weight coefficient of the target achievement degree; is a current drawing speed, is an instantaneous energy consumption, and β and γ are weight coefficients corresponding to the two terms, respectively.

8. A reinforcement learning based intelligent optimization system for ultra-fine tungsten wire drawing process, characterized in that, It includes: A multi-modal data acquisition unit is configured on the drawing production line for real-time acquisition of multi-modal state data as claimed in claim 2; An edge computing decision unit is in communication connection with the data acquisition unit, and internally deployed with a pre-trained hierarchical reinforcement learning strategy model and a forward-looking safety assessment module as claimed in claim 1, for receiving state data and generating final action parameters; A device control execution unit is used to receive the final action parameters and convert them into bottom-level control instructions for drawing equipment and online heat treatment equipment.

9. The system of claim 8, wherein, It also includes a cloud training and digital twin platform, which includes: A digital twin module is used to simulate the drawing process with high fidelity according to the method of claim 6; A model training module is used to pre-train, verify and iteratively update the hierarchical reinforcement learning strategy model in the digital twin module; A model management and deployment module is used to distribute the mature model trained in the cloud to the edge computing decision unit.

10. A computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images