Method and system for optimizing automobile part machining process in combination with reinforcement learning

Abstract

The invention provides an automobile part machining process optimization method and system combined with reinforcement learning. The method comprises the following steps: firstly, acquiring an initial machining process data set comprising process parameters, equipment operation states and product quality detection data; then, a reinforcement learning process optimization environment and an intelligent agent structure are constructed based on the reinforcement learning, an environment simulation processing scene is performed, and an intelligent agent generates an adjustment strategy; then the intelligent agent is controlled to interact with the environment, the intelligent agent generates an adjustment scheme according to an environment feedback state, and the environment outputs a reward signal; evaluating processing quality, efficiency and cost indexes for the adjustment scheme in combination with a reward signal; and finally, according to an evaluation result, adjusting agent strategy parameters, updating an environment, and repeating interactive evaluation until a scheme meeting an optimization target is obtained and output, so that intelligent optimization of the automobile part processing technology is realized.

Description

A Method and System for Optimizing Automotive Parts Manufacturing Processes Using Reinforcement Learning Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for optimizing the processing technology of automotive parts by incorporating reinforcement learning. Background Technology

[0002] The processing technology of automotive parts plays a crucial role in product quality, production efficiency, and cost control. Traditional methods for optimizing automotive parts processing technology mainly rely on the accumulated experience of engineers and extensive experimental testing. Engineers adjust process parameters based on their professional knowledge and past experience; however, this approach has significant limitations. On the one hand, engineers' experience is subjective and limited, making it difficult to fully consider the interactions between various complex factors during processing, potentially leading to incomplete and inaccurate optimization solutions. On the other hand, extensive experimental testing not only consumes significant time, manpower, and material resources, but also carries the risk of uncontrollable factors affecting the accuracy and reliability of experimental results. Furthermore, with the rapid development of the automotive industry, the requirements for the quality and production efficiency of parts are constantly increasing. Traditional optimization methods struggle to adapt quickly to market changes and new production demands, failing to provide timely and effective process optimization solutions, thus limiting the development of automotive parts processing technology and the enhancement of product competitiveness. Summary of the Invention

[0003] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for optimizing the processing technology of automotive parts by incorporating reinforcement learning. The method includes: acquiring an initial processing technology data set, the initial processing technology data set including process parameter data, equipment operating status data, and product quality inspection data during the processing of automotive parts; constructing a reinforcement learning-based process optimization environment and an intelligent agent structure based on the initial processing technology data set, the process optimization environment simulating processing technology execution scenarios, and the intelligent agent structure generating process adjustment strategies; controlling the interaction between the intelligent agent structure and the process optimization environment, the intelligent agent structure generating process adjustment schemes based on status information fed back by the process optimization environment, and the process optimization environment outputting a corresponding set of reward signals based on the process adjustment schemes; performing process effect evaluation processing on the process adjustment schemes in conjunction with the corresponding reward signals, determining whether the processing quality, efficiency, and cost indicators corresponding to each process adjustment scheme meet preset requirements, and obtaining a process effect evaluation result; adjusting the strategy parameters of the intelligent agent structure based on the process effect evaluation result, updating the process optimization environment based on the adjusted intelligent agent structure, repeating the interaction and evaluation process until a process adjustment scheme that meets the preset optimization objective is obtained, and outputting the final optimized processing technology scheme for automotive parts.

[0004] In another aspect, embodiments of the present invention also provide an automotive parts processing optimization system incorporating reinforcement learning, comprising a processor and a machine-readable storage medium connected to the processor. The machine-readable storage medium is used to store programs, instructions, or code, and the processor is used to run the programs, instructions, or code in the machine-readable storage medium to implement the above-described method.

[0005] Based on the above, this embodiment of the invention acquires an initial processing technology data set including process parameter data, equipment operating status data, and product quality inspection data. Based on this data set, a reinforcement learning-based process optimization environment and intelligent agent structure are constructed. The process optimization environment can accurately simulate the processing technology execution scenario, while the intelligent agent structure can autonomously generate process adjustment strategies, achieving intelligent modeling of the processing technology. During the interaction between the intelligent agent structure and the process optimization environment, the intelligent agent generates a process adjustment plan based on the status information fed back by the environment. The environment outputs a set of reward signals based on the adjustment plan. This interaction mechanism enables the intelligent agent to continuously learn and adapt to changes in the processing technology. The process adjustment plan is evaluated in conjunction with the reward signals to determine whether the processing quality, efficiency, and cost indicators meet preset requirements. Based on the evaluation results, the strategy parameters of the intelligent agent structure are adjusted and the process optimization environment is updated. Through repeated interaction and evaluation processes, the intelligent agent can continuously optimize the process adjustment plan, ultimately outputting a final automotive parts processing technology optimization plan that meets the preset optimization goals, effectively improving the quality, efficiency, and cost control level of automotive parts processing technology. Attached Figure Description

[0006] Figure 1 is a schematic diagram of the execution flow of the automotive parts processing optimization method combined with reinforcement learning provided in an embodiment of the present invention.

[0007] Figure 2 is a schematic diagram of the hardware architecture of the automotive parts processing optimization system combined with reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0008] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 is a flowchart illustrating an embodiment of the present invention of a method for optimizing the processing technology of automotive parts using reinforcement learning. The following is a detailed description of this method for optimizing the processing technology of automotive parts using reinforcement learning.

[0009] Step S110: Obtain an initial processing technology data set, which includes process parameter data, equipment operating status data, and product quality inspection data during the processing of automotive parts.

[0010] This embodiment uses the optimization of the crankshaft machining process for automotive engines as a scenario. Crankshaft machining involves multiple processes such as forging, turning, milling, and grinding, requiring data acquisition and integration to form an initial machining process data set. First, a data acquisition system is built, covering three major categories of data acquisition modules: process parameters, equipment status, and quality inspection, ensuring that the data covers the entire crankshaft machining process.

[0011] Step S111: Collect process parameter data corresponding to each process in the processing of automotive parts. The process parameter data includes processing temperature data, processing pressure data, tool speed data, and processing feed data. Record the process parameter data when each process is executed.

[0012] For different processes in crankshaft machining, corresponding process parameter acquisition sensors are configured: In the forging process, a temperature sensor is installed near the mold of the forging equipment to collect machining temperature data, and a pressure sensor is installed on the pressure actuator to collect machining pressure data; In the turning process, a speed sensor is installed on the spindle of the CNC lathe to collect tool speed data, and a displacement sensor is installed on the feed slide to collect machining feed data; The same parameter acquisition logic is used for the milling and grinding processes to ensure that the core process parameters of each process can be acquired in real time.

[0013] The data acquisition system synchronously records parameter data according to the execution sequence of the processes. Each data entry includes a process identifier (e.g., "Forging-1", "Turning-2"), a data acquisition timestamp, a parameter name, and a parameter value. For example, a data entry for the forging process might be "Process: Forging-1, Time: XXX, Parameter: Machining Temperature, Value: XXX; Parameter: Machining Pressure, Value: XXX," while a data entry for the turning process might be "Process: Turning-2, Time: XXX, Parameter: Tool Speed, Value: XXX; Parameter: Machining Feed Rate, Value: XXX." The acquired data is transmitted to the local data server in real time to prevent data loss.

[0014] Step S112: Collect equipment operating status data during the operation of the processing equipment. The equipment operating status data includes equipment motor operating current data, equipment transmission component vibration frequency data, equipment cooling system flow data, and equipment component temperature data. The equipment operating status data is acquired and stored in real time.

[0015] Condition monitoring modules are deployed on forging equipment, CNC lathes, milling machines, grinding machines, and other equipment used in crankshaft processing: current sensors are installed on the power supply lines of the equipment motors to collect operating current data; vibration sensors are installed on the transmission components such as the spindle and gearbox to collect vibration frequency data; flow sensors are installed on the pipes of the equipment cooling system to collect cooling system flow data; and temperature sensors are installed on key components such as the spindle box and guide rails to collect temperature data of each component.

[0016] The status monitoring module collects data at fixed time intervals (e.g., once per second). Each data entry includes a unique equipment identifier (e.g., "Forging Equipment-F01", "CNC Lathe-C02"), component name, parameter name, and parameter value. For example, a status data record for the CNC lathe might be "Equipment: C02, Component: Spindle Motor, Parameter: Operating Current, Value: XXX; Parameter: Temperature, Value: XXX," while a data record for the cooling system might be "Equipment: C02, Component: Cooling Pipe, Parameter: Flow Rate, Value: XXX." The collected equipment operating status data and process parameter data are linked via timestamps to ensure that process parameters and equipment status data for the same process step can be matched.

[0017] Step S113: Obtain the product quality inspection data of the tested automotive parts. The product quality inspection data includes the dimensional accuracy data, surface roughness data, mechanical performance data and defect detection data of the parts. Organize the product quality inspection data corresponding to each batch of products.

[0018] After the crankshaft is machined, quality inspection data is obtained through specialized testing equipment: a coordinate measuring machine is used to check dimensional accuracy data (such as crankshaft main journal diameter and connecting rod journal spacing); a surface roughness tester is used to check surface roughness data (such as main journal surface roughness and flange end face roughness); a universal testing machine is used to check mechanical property data (such as tensile strength and yield strength); and an eddy current flaw detector is used to check defect detection data (such as internal cracks and inclusions).

[0019] After each batch of crankshafts is inspected, the quality inspection data is compiled. Each data entry includes the batch number, unique product identifier, inspection item name, and inspection value. For example, the quality data record for a certain batch of crankshafts is: "Batch: B001, Product: Q001, Inspection Item: Main journal diameter, value: XXX; Item: Surface roughness, value: XXX; Item: Tensile strength, value: XXX; Item: Defect detection, Result: No defects." The quality inspection data is linked to the corresponding batch's processing parameters and equipment operating status data through the batch number, forming a complete product lifecycle data chain.

[0020] Step S114: Classify and integrate the collected process parameter data, equipment operating status data, and product quality inspection data, and associate the data according to the order of processing steps to form a data association mapping relationship.

[0021] To build a data integration platform, process parameter data, equipment operating status data, and product quality inspection data are first categorized and stored in different data tables according to data type. Then, based on the sequence of processes (forging → turning → milling → grinding → inspection), data association mapping relationships are established using key fields such as process identifiers, equipment identifiers, and batch numbers.

[0022] For example, using batch number "B001" as the keyword, the system associates the following data with the batch of crankshafts: process parameters (processing temperature, pressure), operating status data of the forging equipment (motor current, vibration frequency), process parameters (tool speed, feed rate) of the turning process, operating status data of the CNC lathe (spindle temperature, cooling flow rate), and final quality inspection data (dimensional accuracy, surface roughness). The resulting mapping is stored in JSON format, with each mapping record containing the associated keyword, the process parameter data ID for each process, the equipment status data ID, and the quality inspection data ID, ensuring data traceability.

[0023] Step S115: Based on the data association mapping relationship, duplicate and invalid data entries are removed, and data with complete process correspondence is retained to generate an initial processing technology data set containing process parameter data, equipment operating status data and product quality inspection data.

[0024] The data cleaning module processes the associated dataset: First, it identifies duplicate data entries by comparing key fields (such as process identifier, collection timestamp, and parameter values) and deleting identical duplicate data; Second, it identifies invalid data entries by removing data with parameter values ​​outside the reasonable range (such as negative processing temperature or zero tool speed), data missing key fields (such as no process identifier or no parameter value), and data that cannot be associated with quality inspection data.

[0025] Retain data with complete process correspondences, meaning that each process from forging to inspection has corresponding process parameters, equipment status data, and complete quality inspection data entries. Organize the cleaned data by batch number to generate an initial processing technology dataset, store it in a structured database (such as MySQL), and generate a dataset description document recording the data volume, covered processes, data fields, and association rules to facilitate subsequent use by reinforcement learning models.

[0026] Step S120: Construct a reinforcement learning process optimization environment and agent structure based on the initial processing technology data set. The process optimization environment is used to simulate the processing technology execution scenario, and the agent structure is used to generate process adjustment strategies.

[0027] Based on the initial machining process data set, a reinforcement learning framework that fits the actual crankshaft machining scenario is constructed. Through environmental simulation and intelligent agent design, the autonomous generation and optimization of process adjustment strategies are realized, as detailed below.

[0028] Step S121: Extract key variables related to process execution from the initial processing technology data set, determine the state space dimension of the process optimization environment, the state space dimension includes process parameter variable dimension, equipment state variable dimension and quality index variable dimension, and define the value range of each state space dimension.

[0029] The initial machining process data set was traversed to extract key variables affecting crankshaft machining quality, efficiency, and cost: process parameter variables include forging temperature, forging pressure, turning speed, turning feed rate, milling speed, milling feed rate, grinding speed, and grinding feed rate; equipment status variables include forging equipment motor current, spindle vibration frequency, cooling flow rate, CNC lathe spindle temperature, and milling machine guideway temperature; quality index variables include crankshaft journal diameter deviation, surface roughness value, tensile strength value, and defect detection results.

[0030] Each state space dimension has a defined value range, determined based on the historical data distribution in the initial machining process dataset. For example, the forging temperature range is from the minimum to the maximum value in the historical data; the equipment motor current range is 80% to 120% of the equipment's rated current; and the spindle diameter deviation range is the allowable positive and negative deviation range according to quality standards. These state space dimensions and value ranges are compiled into a state space definition document, serving as the foundation for constructing the process optimization environment.

[0031] Step S122: Based on the adjustment requirements of automotive parts processing technology, classify the types of process adjustment actions, including process parameter fine-tuning actions, equipment status calibration actions, and process sequence adjustment actions.

[0032] To address the process optimization needs of crankshaft machining, three types of actions are categorized: Fine-tuning of process parameters, including adjusting forging temperature (± a certain range), forging pressure (± a certain range), tool speed (± a certain range), and feed rate (± a certain range), with each fine-tuning action corresponding to a specific parameter adjustment range; Equipment status calibration, including calibrating motor operating current (restoring it to the rated range), adjusting cooling system flow (increasing or decreasing it), and calibrating transmission component vibration frequency (reducing it to the normal range), proposing calibration measures for abnormal equipment status; and Process sequence adjustment, including adjusting the order of turning and milling, adding a semi-finishing process before grinding, and adjusting the execution frequency of inspection processes, optimizing the process flow.

[0033] Assign a unique identifier to each action type (e.g., “P-001” represents fine-tuning of forging temperature, “E-001” represents motor current calibration, and “S-001” represents process sequence adjustment), and define the execution logic of each action in detail (e.g., the execution logic of “P-001” is “adjust the power of the heating device by a certain step size according to the difference between the current forging temperature and the target temperature”), thus forming a list of action types.

[0034] Step S123: Construct the reward function of the process optimization environment. The input of the reward function is the standardized value of the change in quality indicators, efficiency, and cost after process adjustment. The reward value is calculated according to the preset weight coefficient. When quality is improved, efficiency is increased, and cost is reduced, the reward value is positive, and vice versa.

[0035] The reward function is constructed through the following sub-steps: Step S1231: Determine the input parameter type of the reward function. The input parameters are determined to be the quality index change value, efficiency change value, and cost change value after process adjustment. The quality index change value is the difference between the quality index after process adjustment and the quality index before process adjustment. The efficiency change value is the difference between the processing time before process adjustment and the processing time after process adjustment. The cost change value is the difference between the processing cost before process adjustment and the processing cost after process adjustment.

[0036] The reward function has three input parameters: the quality index change value is selected as the weighted difference of three core indicators: crankshaft main journal diameter deviation, surface roughness, and tensile strength. The calculation method is "weighted value of the index after adjustment - weighted value of the index before adjustment", and a positive value indicates quality improvement; the efficiency change value is "total processing time per batch before adjustment - total processing time per batch after adjustment", and a positive value indicates efficiency improvement; the cost change value is "total processing cost per batch before adjustment - total processing cost per batch after adjustment", and a positive value indicates cost reduction.

[0037] For example, before adjustment, the spindle journal diameter deviation was a certain value, the surface roughness was a certain value, the tensile strength was a certain value, and the weighted value was a certain value; after adjustment, the weighted value of the corresponding index was a certain value, and the change in quality index was the difference between the two; before adjustment, the processing time was a certain value, and after adjustment, it was a certain value, and the change in efficiency was the difference between the two; before adjustment, the cost was a certain value, and after adjustment, it was a certain value, and the change in cost was the difference between the two.

[0038] Step S1232: Determine the weighting coefficients corresponding to each input parameter using expert experience. The weighting coefficients for the quality index change values ​​are set according to the degree of influence of automotive parts quality on automotive product performance. The weighting coefficients for the efficiency change values ​​are set according to the degree of influence of processing efficiency on the enterprise's production plan. The weighting coefficients for the cost change values ​​are set according to the degree of influence of processing cost on the enterprise's production efficiency. The sum of the weighting coefficients for the quality index change values, the efficiency change values, and the cost change values ​​is 1.

[0039] A review panel composed of crankshaft machining experts, production management experts, and cost accounting experts was invited to determine the weighting coefficients using the Delphi method: considering that crankshaft quality directly affects engine performance, the weighting coefficient for changes in quality indicators was set to a certain value; machining efficiency affects production delivery cycle, so the weighting coefficient for changes in efficiency was set to a certain value; and cost affects corporate profit, so the weighting coefficient for changes in cost was set to a certain value. The sum of the three was 1.

[0040] For example, the quality weighting coefficient is 0.5, the efficiency weighting coefficient is 0.3, and the cost weighting coefficient is 0.2, ensuring that quality plays a dominant role in the optimization objective while also taking into account efficiency and cost.

[0041] Step S1233: Construct the mathematical expression of the reward function. The reward value is obtained by standardizing and adding the products of the quality indicator change value and the weight coefficient of the quality indicator change value, the efficiency change value and the weight coefficient of the efficiency change value, and the cost change value and the weight coefficient of the cost change value.

[0042] First, the three input parameters are standardized to convert the parameter values ​​to the range of [-1, 1]. The standardization method is "(parameter value - minimum parameter value) / (maximum parameter value - minimum parameter value) × 2 - 1", where the minimum and maximum parameter values ​​are determined based on historical data in the initial processing technology dataset.

[0043] The reward function expression is: "Reward Value = (Standardized Value of Quality Indicator Change × Quality Weight Coefficient) + (Standardized Value of Efficiency Change × Efficiency Weight Coefficient) + (Standardized Value of Cost Change × Cost Weight Coefficient)". For example, if the standardized value of quality change is 0.8, efficiency is 0.6, and cost is 0.4, the reward value = 0.8 × 0.5 + 0.6 × 0.3 + 0.4 × 0.2 = 0.4 + 0.18 + 0.08 = 0.66, which is a positive value indicating that the process adjustment plan is effective.

[0044] Step S1234: Set the adjustment rules for the reward value. When the change value of the quality indicator is positive, the change value of the efficiency is positive, and the change value of the cost is positive, the reward value is positive. When the change value of any input parameter is negative, the reward value is reduced accordingly according to the weight coefficient of the change value of that input parameter. If the change values ​​of two or three input parameters are negative, the reward value is negative.

[0045] Detailed rules for adjusting reward values ​​are established: If the changes in all three parameters are positive, the reward value is calculated normally according to the expression; if the change in quality is positive, the change in efficiency is positive, and the change in cost is negative, the reward value = (standardized quality value × quality weight) + (standardized efficiency value × efficiency weight) - (|standardized cost value| × cost weight); if the change in quality is negative, the reward value is negative regardless of other parameters, and the negative weight is doubled (e.g., if the standardized quality value is -0.5, the corresponding item in the reward value is -0.5 × 0.5 × 2 = -0.5), emphasizing the principle of prioritizing quality.

[0046] For example, if the standardized value of quality change is 0.7, efficiency is 0.5, and cost is -0.3, the reward value = 0.7×0.5+0.5×0.3-0.3×0.2=0.35+0.15-0.06=0.44, which is still a positive value but lower than the reward value when all three are positive; if the standardized value of quality change is -0.4, efficiency is 0.6, and cost is 0.3, the reward value = -0.4×0.5×2+0.6×0.3+0.3×0.2=-0.4+0.18+0.06=-0.16, which is a negative value.

[0047] Step S1235: Verify the reward function by selecting a portion of historical process adjustment data from the initial processing technology data set, inputting the changes in quality indicators, efficiency, and cost after the historical process adjustments into the reward function, calculating the reward value, comparing the calculated reward value with the actual effect of the historical process adjustments, and determining whether the reward value can accurately reflect the actual effect of the historical process adjustments.

[0048] One hundred historical process adjustment data points were selected from the initial processing technology dataset, covering three types: effective adjustments, ineffective adjustments, and negative adjustments. The quality, efficiency, and cost changes of each data point were input into a reward function to calculate the corresponding reward value. Then, the reward value was compared with the actual effects of the historical records (e.g., "quality improved, efficiency increased" or "quality decreased, cost increased").

[0049] The matching degree between the statistical reward value and the actual effect is determined. If the reward value for positive adjustment (good actual effect) is positive and the reward value for negative adjustment (poor actual effect) is negative, and the matching degree reaches 90% or more, the reward function is deemed effective. If the matching degree is less than 90%, the reasons for the deviation are analyzed, such as unreasonable weight coefficients or inappropriate standardization methods.

[0050] Step S1236: Adjust the weight coefficients of each input parameter change value according to the verification results. If the deviation between the reward value and the actual effect of the historical process adjustment is greater than the set deviation threshold, adjust the weight coefficients of the corresponding input parameter change value. Repeat the verification and adjustment process until the reward value can accurately reflect the comprehensive impact of the process adjustment on quality, efficiency and cost, and determine the final reward function.

[0051] If the verification finds that the reward value for some positive adjustments is too low (such as an adjustment that significantly improves quality but slightly decreases efficiency, resulting in a negative reward value), then the efficiency weight coefficient should be appropriately reduced and the quality weight coefficient increased. If the reward value for some negative adjustments is too high (such as an adjustment that decreases quality but reduces costs, resulting in a positive reward value), then the quality weight coefficient should be further increased and the cost weight coefficient decreased.

[0052] After adjustment, the verification is repeated multiple times until the matching degree between the reward value and the actual effect reaches more than 95%. The final reward function is determined, including the standardization method, weight coefficient and adjustment rules, and stored as a function configuration file for use in the process optimization environment.

[0053] Step S124: Construct the policy network structure of the intelligent agent structure. The policy network structure includes an input layer, a hidden layer and an output layer. The input layer receives the state information of the process optimization environment. The hidden layer processes the state information through a fully connected layer and an activation function. The output layer outputs the probability distribution of each process adjustment action.

[0054] The policy network adopts a fully connected neural network structure: the number of nodes in the input layer is the same as the number of dimensions in the state space, and each node corresponds to a state variable (such as forging temperature, motor current, and main journal diameter deviation), receiving the state vector output by the process optimization environment; the hidden layer contains two fully connected layers, the number of nodes in the first hidden layer is twice that of the input layer, and the number of nodes in the second hidden layer is half that of the first layer; the number of nodes in the output layer is the same as the number of action types, and each node corresponds to a process adjustment action (such as temperature fine-tuning and current calibration).

[0055] The input layer converts the state vector into a numerical form that the neural network can process. This involves normalizing each variable value in the state vector to a preset range, ensuring that state variables of different dimensions have a uniform numerical range to meet the input requirements of the neural network. The input layer passes the normalized state vector to the first hidden layer via a weight matrix. The first hidden layer performs a linear transformation on the input vector, mapping the original state dimension to a higher-dimensional feature space, generating the first feature vector to capture the fundamental relationships between state variables.

[0056] Subsequently, the first feature vector is input into the ReLU activation function, which suppresses negative values ​​in the vector, retaining only positive features to generate an activated second feature vector. This introduces nonlinearity to enhance the network's ability to express complex features. The second feature vector is then passed to the second hidden layer, where the linear transformation and ReLU activation operations are repeated to further extract higher-order features, such as the nonlinear relationship between forging temperature and main journal diameter deviation, and the coordinated variation of motor current and cooling flow rate.

[0057] After processing through two hidden layers, the resulting high-level feature vector is input to the output layer. The output layer uses the softmax activation function to normalize the high-level feature vector and calculates the probability value corresponding to each process adjustment action. The sum of the probability values ​​of all actions is 1, forming a complete action probability distribution. For example, the output layer generates corresponding probability distributions for actions such as "fine-tuning forging temperature," "calibrating motor current," and "adjusting process sequence." The action with the highest probability value will be the priority choice for the agent.

[0058] Step S125: Configure the value network structure of the agent structure. The value network structure is used to evaluate the expected value of each action in the current state. The input is state information and action information, and the output is the expected reward value. Integrate the policy network structure and the value network structure to form a complete agent structure.

[0059] The value network employs a fully connected neural network structure. Its input layer contains two parts of data: one part is a state information vector consistent with the policy network input, covering all state variables such as process parameters, equipment status, and quality indicators; the other part is an action information vector represented by one-hot encoding, where each action type corresponds to an independent dimension, the selected action is the preset activation value, and the remaining actions are zero values. The two vector parts are concatenated to form the input vector of the value network, and the number of input layer nodes is the sum of the dimension of the state vector and the number of action types.

[0060] The hidden layer of the value network consists of two fully connected layers. The first layer has twice the number of nodes as the input layer, while the second layer has the same number of nodes as the first layer. Both layers use the ReLU activation function to process the feature vectors. Through layer-by-layer processing, high-order correlation features of state-action combinations are gradually extracted, such as the quality improvement potential and cost change trend corresponding to "fine-tuning of forging temperature + current motor current state".

[0061] The output layer is a single-node structure, outputting the expected reward value corresponding to the current state-action combination. This expected reward value comprehensively reflects the short-term and long-term cumulative rewards that may be obtained after performing the action, providing a valuable reference for the action selection of the policy network. To improve training efficiency and feature consistency, the policy network and the value network share the weight parameters of the first hidden layer, reducing the number of network parameters while ensuring that the extraction logic of basic features is consistent between the two. The integrated agent structure realizes real-time interaction between policy generation and value evaluation through an internal data interface, collaboratively optimizing the process and adjusting the policy.

[0062] Step S126: Associate the state space dimension, action type, reward function with the agent structure to generate a process optimization environment and agent structure that simulates the execution scenario of the processing technology and supports the interaction of the agent structure for reinforcement learning.

[0063] The core simulation engine for building the process optimization environment integrates the state space defined in step S121, the action types divided in step S122, and the reward function determined in step S123 into the simulation engine. Based on the historical process execution logic in the initial machining process data set, the simulation engine constructs a process-level simulation model for crankshaft machining. This process-level simulation model includes the correlation logic between process parameters, equipment status, and quality indicators for each process such as forging, turning, milling, and grinding. It can adjust actions according to the input process and simulate the changes in parameters, equipment status response, and the evolution of quality indicators after adjustment.

[0064] An interaction interface is established between the environment and the agent: the agent sends process adjustment commands to the environment through the interface. After receiving the commands, the environment calls the simulation model to simulate the execution of the actions, updating the values ​​of various variables in the state space, including the adjusted process parameters, equipment operating status, and corresponding quality indicators. Subsequently, the environment calls the reward function to calculate the reward value based on the updated state variables, and then feeds back the updated state information and reward value to the agent through the interaction interface.

[0065] For example, after the agent sends the "forging temperature fine-tuning" command, the environment simulation increases the forging temperature. Based on historical data correlation patterns, it updates the corresponding equipment state variables such as motor current and cooling flow rate, as well as quality and efficiency indicators such as spindle diameter deviation and processing time. The reward value corresponding to this adjustment is calculated using a reward function. Finally, the new temperature value, current value, diameter deviation value, and reward value are fed back to the agent. After completing the interactive interface debugging, a complete reinforcement learning framework containing the process optimization environment and agent structure is generated, and a framework configuration file is output to record the core parameters.

[0066] Step S130: Control the intelligent agent structure to interact with the process optimization environment. The intelligent agent structure generates a process adjustment plan based on the status information fed back by the process optimization environment. The process optimization environment outputs a corresponding set of reward signals based on the process adjustment plan.

[0067] The interactive training process of the reinforcement learning framework is initiated. Through continuous dynamic interaction between the agent and the environment, multiple sets of process adjustment schemes are generated and corresponding reward signals are obtained. The specific process is as follows.

[0068] Step S131: Initialize the process optimization environment by taking the initial process parameter data, initial equipment operating status data, and initial product quality inspection data in the initial processing process data set as the initial state information of the process optimization environment and sending the initial state information to the intelligent agent structure.

[0069] Complete machining data for a batch of crankshafts is selected from the initial machining process data set. The initial process parameters (covering the core parameters of each process of forging, turning, milling, and grinding), initial equipment operating status (motor current, vibration frequency, cooling flow, etc. of each machining equipment), and initial product quality inspection data (main journal diameter deviation, surface roughness, tensile strength, etc.) of this batch are extracted as the initial state of the process optimization environment.

[0070] The initial state information is converted into a state vector that meets the input requirements of the agent. The vector contains process parameter sub-vectors, equipment state sub-vectors, and quality index sub-vectors. It is sent to the policy network input layer of the agent structure through the interaction interface to trigger the agent's action generation process.

[0071] Step S132: Receive the initial state information through the agent structure, input it into the policy network structure of the agent structure, process the initial state information through the hidden layer of the policy network structure of the agent structure, generate the probability distribution of each process adjustment action through the output layer of the policy network structure of the agent structure, select the process adjustment action with the highest probability according to the probability distribution, and combine them to form an initial process adjustment scheme.

[0072] The initial process adjustment plan is generated through the following sub-steps: Step S1321: The initial state information sent by the process optimization environment is received through the input layer of the policy network structure of the intelligent agent structure, and converted into a vector form that can be processed by the policy network structure of the intelligent agent structure. The initial state information includes initial process parameter data, initial equipment operating status data and initial product quality inspection data.

[0073] After receiving the initial state vector from the environment, the input layer of the agent normalizes each variable value in the vector to a preset range through a data transformation module. This ensures that different types of state variables (such as temperature, current, and diameter deviation) have a uniform numerical range, adapting to the processing requirements of the policy network. The transformed state vector fully retains information on the initial process parameters, equipment status, and quality indicators, with a total dimension consistent with the number of nodes in the policy network's input layer.

[0074] Step S1322: Input the transformed state vector into the hidden layer of the policy network structure of the agent structure. The hidden layer of the policy network structure of the agent structure contains multiple fully connected layers. The first fully connected layer of the hidden layer of the policy network structure of the agent structure performs a linear transformation on the state vector and maps the state vector to a higher-dimensional feature space through the weight matrix to obtain the first feature vector.

[0075] The transformed state vector is input into the first hidden layer of the policy network. This first hidden layer performs a linear transformation on the state vector using a preset weight matrix, mapping the original state dimension to a higher-dimensional feature space and generating the first feature vector. This process can capture the basic linear relationship between state variables, such as the proportional relationship between forging pressure and equipment motor current.

[0076] Step S1323: Input the first feature vector into the activation function, which uses the ReLU function to suppress the negative values ​​in the first feature vector and retain the positive features in the first feature vector to generate the activated second feature vector.

[0077] The first feature vector is input to the ReLU activation function, which evaluates each element in the first feature vector. If the element value is negative, it is set to zero; if it is positive, it is kept as is, generating the activated second feature vector. This operation suppresses irrelevant or redundant feature components, enhances the expression of effective features, and introduces nonlinear characteristics to adapt to complex process optimization scenarios.

[0078] Step S1324: Input the second feature vector into the next fully connected layer of the hidden layer of the policy network structure of the agent structure, repeat the linear transformation and activation operation, and after multiple layers of processing by the hidden layer of the policy network structure of the agent structure, input the final high-level feature vector into the output layer of the policy network structure of the agent structure.

[0079] The second feature vector is then passed to the second hidden layer, where linear transformation and ReLU activation operations are repeatedly performed to further extract higher-order nonlinear correlation features between state variables, such as the combined effect of the "forging temperature + turning speed" combination on the surface roughness of the main journal. After processing through two hidden layers, the resulting high-level feature vector contains abstract correlation rules of state information and is input to the output layer of the policy network.

[0080] Step S1325: The output layer of the policy network structure of the intelligent agent structure is a softmax layer. The high-level feature vectors are normalized, and the probability value corresponding to each process adjustment action is calculated. The sum of the probability values ​​of all process adjustment actions is 1, forming the probability distribution of each process adjustment action.

[0081] After the high-level feature vectors are input to the output layer, the softmax function processes each element in the vector. Through exponential and normalization operations, each element is converted into a probability value for the corresponding process adjustment action. The sum of the probability values ​​of all actions is 1, forming a complete action probability distribution. This distribution reflects the priority of each process adjustment action in the current state.

[0082] Step S1326: Extract the probability value of each process adjustment action from the probability distribution of each process adjustment action, compare the probability of all process adjustment actions, and select the process adjustment action with the highest probability. If there are multiple process adjustment actions with the same and highest probability, then determine one of the process adjustment actions by random selection.

[0083] The algorithm iterates through the probability distribution of actions generated by the output layer, extracts the probability value corresponding to each process adjustment action, and determines the action with the highest probability by comparison. If multiple actions have the same highest probability, a random number generation mechanism is used to select one of these actions to avoid decision-making stagnation caused by equal probabilities.

[0084] Step S1327: Based on the selected process adjustment actions and the sequence of automotive parts processing, determine the adjustment range of process parameters, equipment operation status adjustment method, and process execution details corresponding to the process adjustment action, and integrate them to form an initial process adjustment plan.

[0085] For the selected process adjustment actions, refer to the corresponding parameter values ​​in the current state and historical process adjustment experience to determine the specific adjustment range (such as the specific range of process parameter fine-tuning), adjustment method (such as which part of the equipment is adjusted to achieve parameter change) and execution timing (such as adjusting before or during a certain process).

[0086] For example, if the selected action is "fine-tuning of forging temperature," then based on the current forging temperature value, the adjustment range is determined to be a preset range, the adjustment method is to change the temperature by controlling the power of the heating device, and the execution time is the preheating stage before the formal start of the forging process. Integrating the action type, adjustment range, adjustment method, and execution time forms a complete initial process adjustment plan.

[0087] Step S133: Input the initial process adjustment scheme into the process optimization environment. The process optimization environment simulates and executes the initial process adjustment scheme, updates process parameter data and equipment operating status data, and calculates the corresponding product quality inspection data based on preset quality assessment rules.

[0088] After receiving the initial process adjustment plan through the interactive interface, the process optimization environment starts the simulation execution module and updates the values ​​of the corresponding process parameters according to the adjustment requirements in the plan. Subsequently, based on the parameter-state correlation patterns recorded in the initial processing technology data set, it simulates the equipment's response to the process parameter adjustment and updates the equipment's operating status data, such as changes in motor current, vibration frequency fluctuations, and adaptive adjustments of cooling flow caused by the process parameter adjustment.

[0089] Finally, based on the preset quality assessment rules (which are derived from historical quality inspection data and reflect the mapping relationship between process parameters and quality indicators), the adjusted product quality inspection data, such as changes in dimensional accuracy, surface roughness, and mechanical properties, are calculated. The entire simulation process strictly follows the actual processing logic to ensure that changes in parameters, states, and quality indicators are reasonable and consistent.

[0090] Step S134: Based on the updated process parameter data, equipment operating status data, and product quality inspection data, calculate the corresponding reward signal through the reward function of the process optimization environment. The reward signal reflects the impact of the initial process adjustment scheme on processing quality, efficiency, and cost.

[0091] Extract data on changes in quality indicators, efficiency, and costs from the simulation results: Changes in quality indicators are reflected in the difference between the adjusted quality inspection data and the data before the adjustment, with positive values ​​indicating improved quality; changes in efficiency are reflected in the difference between the adjusted processing time and the data before the adjustment, with positive values ​​indicating improved efficiency; changes in costs are reflected in the difference between the adjusted processing cost and the data before the adjustment, with positive values ​​indicating reduced costs.

[0092] The three sets of changed data are input into the reward function. First, the data is standardized using a preset method to transform it into a uniform range. Then, based on preset weighting coefficients, the three standardized data sets are weighted and calculated to obtain the reward value. The sign and magnitude of the reward value directly reflect the comprehensive impact of the process adjustment scheme on quality, efficiency, and cost.

[0093] Step S135: The calculated reward signal and updated state information are fed back to the intelligent agent structure through the process optimization environment. The intelligent agent structure receives the new state information and repeats the above process of generating the process adjustment scheme, executing the process adjustment scheme in the process optimization environment, and calculating the reward signal.

[0094] The process optimization environment sends reward signals and updated status information (including adjusted process parameters, equipment status, and quality indicators) to the agent through an interactive interface. After receiving the new status information, the agent repeats the action generation process of step S132, generates the next process adjustment plan based on the new status, and the environment executes the plan again and feeds back reward signals, forming a continuous interactive loop.

[0095] Step S136: Record the process adjustment schemes and corresponding reward signals generated in each interaction, and organize them into a set of process adjustment schemes and a set of corresponding reward signals according to the interaction order.

[0096] During the interaction, each time a cycle of "agent generation scheme - environment execution - feedback reward" is completed, the process adjustment scheme and the corresponding reward signal for that interaction are recorded. After the number of interactions reaches a preset number of rounds, all records are organized in the order of interaction to form a set of process adjustment schemes and a set of reward signals, with each scheme associated with a corresponding reward signal.

[0097] Step S140: The process adjustment scheme is combined with the corresponding reward signal to perform process effect evaluation processing, and it is determined whether the processing quality, efficiency and cost indicators corresponding to each process adjustment scheme meet the preset requirements, so as to obtain the process effect evaluation result.

[0098] Based on the set of process adjustment schemes and reward signals, each scheme is comprehensively evaluated from three core dimensions: quality, efficiency, and cost, and the schemes that meet the optimization objectives are selected, as detailed below.

[0099] Step S141: Extract the process parameter adjustment content, equipment operation status adjustment content, and process adjustment content corresponding to each process adjustment scheme from the set of process adjustment schemes.

[0100] The process adjustment scheme set is traversed, and each scheme is analyzed to extract details of process parameter adjustments (such as the type, direction, and magnitude of adjustments), equipment operating status adjustment requirements (such as equipment components requiring calibration and the target state after adjustment), and process adjustment content (such as changes in process sequence, additions, or deletions of processes). The analysis results are categorized and stored according to scheme number to form a list of scheme adjustment content.

[0101] Step S142: Based on the reward signals in the reward signal set that correspond one-to-one with the process adjustment schemes, analyze the quality index change value, efficiency change value, and cost change value corresponding to each process adjustment scheme. The quality index change value corresponds to the improvement or decrease in product quality, the efficiency change value corresponds to the shortening or extension of the processing cycle, and the cost change value corresponds to the reduction or increase in processing cost.

[0102] Each reward value in the reward signal set is associated with the original data that generated that reward value. By parsing the calculation log of the reward function, the change values ​​of quality indicators, efficiency, and cost corresponding to each process adjustment scheme are extracted. These change values ​​directly reflect the actual effects of the scheme on quality improvement, efficiency enhancement, and cost control after its implementation.

[0103] Step S143: Set preset requirements for processing quality, efficiency and cost. The preset requirements for processing quality include the allowable deviation range of dimensional accuracy, the allowable range of surface roughness and the standard for mechanical properties. The preset requirements for efficiency include the upper limit of processing time per batch. The preset requirements for cost include the lower limit of processing cost per piece.

[0104] Referring to national standards, industry specifications, and internal production requirements for crankshaft machining in the automotive industry, preset requirements for quality, efficiency, and cost are set: Regarding quality, the permissible deviation range for dimensional accuracy, the upper limit standard for surface roughness, and the threshold values ​​for mechanical properties (such as tensile strength and yield strength) are clearly defined; regarding efficiency, an upper limit for the machining time of a single batch of crankshafts is set; regarding cost, an upper limit for the cost of machining a single crankshaft is set. These preset requirements are compiled into standardized evaluation criteria to serve as the basis for performance evaluation.

[0105] Step S144: Compare the changes in quality indicators, efficiency, and cost corresponding to each process adjustment scheme with the corresponding preset requirements to determine whether the quality indicators meet the allowable deviation range of dimensional accuracy, the allowable range of surface roughness, and the mechanical performance standards; whether the efficiency meets the upper limit of single batch processing time; and whether the cost meets the lower limit of single piece processing cost.

[0106] For each process adjustment plan, the actual values ​​of the corresponding quality indicators after the changes are compared with the preset requirements to determine whether the dimensional accuracy is within the allowable deviation, whether the surface roughness is below the upper limit, and whether the mechanical properties meet the standards. The adjusted single-batch processing time is compared with the upper limit of the preset efficiency requirements to determine whether the efficiency requirements are met. The adjusted single-piece processing cost is compared with the upper limit of the preset cost requirements to determine whether the cost control requirements are met.

[0107] Step S145: Count the number of indicators that meet the preset requirements in each process adjustment plan. If the three indicators of quality, efficiency and cost all meet the preset requirements, the process adjustment plan is marked as a qualified process adjustment plan. If two indicators meet the preset requirements, the process adjustment plan is marked as a process adjustment plan to be improved. If less than two indicators meet the preset requirements, the process adjustment plan is marked as a unqualified process adjustment plan.

[0108] Based on the comparison results, the number of indicators that each solution meets the preset requirements is counted: if all three indicators meet the requirements, it indicates that the solution has excellent overall effect and is marked as "qualified"; if two indicators meet the requirements, it indicates that the solution has certain defects but has the potential for improvement and is marked as "needs improvement"; if only one or zero indicators meet the requirements, it indicates that the solution is not effective and is marked as "unqualified".

[0109] Step S146: Organize the marking results of all process adjustment schemes, and combine the changes in quality indicators, efficiency, and cost corresponding to each process adjustment scheme to form a process effectiveness evaluation result. This process effectiveness evaluation result is presented in a structured document format, with each record including the scheme number, a summary of the adjustment content, specific changes in quality / efficiency / cost, the number of compliant indicators, and the marking result (qualified / needs improvement / unqualified).

[0110] For example, a scheme numbered A001 has the adjustment content of "fine-tuning of forging temperature + optimization of cooling flow". The quality indicators change to "improvement of diameter deviation and reduction of roughness" (both meet the standards), the efficiency change is "shortening of processing time" (meeting the standards), and the cost change is "reduction of energy consumption" (meeting the standards). The number of met indicators is 3, and the result is marked as "qualified". A scheme numbered A002 has the adjustment content of "increasing turning speed". The quality indicator change is "roughness exceeds the standard" (not met the standards), the efficiency change is "shortening of processing time" (meeting the standards), and the cost change is "increased tool wear leading to increased costs" (not met the standards). The number of met indicators is 1, and the result is marked as "unqualified".

[0111] Meanwhile, a statistical analysis section is added to the evaluation results to summarize the proportion of qualified solutions, solutions requiring improvement, and unqualified solutions, and to analyze the commonalities of adjustment actions for various solutions (such as qualified solutions often involving coordinated adjustments of temperature and cooling), providing guidance for subsequent adjustments to agent parameters.

[0112] Step S150: Adjust the strategy parameters of the intelligent agent structure according to the process effect evaluation results, update the process optimization environment based on the adjusted intelligent agent structure, repeat the interaction and evaluation process until a process adjustment scheme that meets the preset optimization target is obtained, and output the final automotive parts processing process optimization scheme.

[0113] By iteratively optimizing, the ability of the intelligent agent to generate high-quality solutions is continuously improved, and finally, a process optimization solution that meets production requirements is output. The specific process is as follows.

[0114] Step S151: Select qualified process adjustment schemes and corresponding reward signals from the process effect evaluation results, and store the adjustment actions of qualified process adjustment schemes, the status information before and after the execution of qualified process adjustment schemes, and the reward signals corresponding to qualified process adjustment schemes into an experience sample and store it in the experience playback buffer.

[0115] The process evaluation results are iterated through, and all process adjustment schemes marked as "qualified" are extracted. The type and magnitude of the adjustment action for each qualified scheme are recorded, along with the initial state information before execution and the target state information after execution, while also associating the corresponding reward signal. This information is combined into an experience sample, with each sample containing a quadruple of "initial state - adjustment action - reward signal - target state".

[0116] All experience samples are stored in the experience replay buffer. The buffer adopts a first-in, first-out storage strategy. When the number of samples reaches the upper limit of the buffer capacity, the earliest stored sample is automatically deleted to ensure that the samples in the buffer are always the latest qualified solution experience, providing high-quality training data for agent parameter optimization.

[0117] Step S152: Randomly select multiple experience samples from the experience replay buffer, input the state information of each experience sample into the value network structure of the agent structure, calculate the expected value of each action in the current state, and simultaneously input the next state information of each experience sample into the value network structure of the agent structure to calculate the expected value of the next state.

[0118] A batch of experience samples is randomly drawn from the experience replay buffer (e.g., several samples are drawn each time) to avoid the influence of sample order on training. For each sample, the initial state information is input into the value network. The value network extracts state features and outputs the expected value of all possible adjustment actions in the current state through linear transformation and activation operations of the hidden layers. Then, the target state information (i.e., the next state) in the sample is input into the value network, and the expected value in that state is calculated in the same way. This expected value reflects the potential for subsequent rewards that can be obtained after performing the adjustment action to enter the new state.

[0119] Step S153: Based on the reward signal, the expected value of the current state, and the expected value of the next state in each experience sample, construct a loss function. The loss function is used to measure the deviation between the predicted value of the value network structure of the agent structure and the actual reward signal and the value of the next state.

[0120] The loss function is constructed as follows: using the expected value of the current state as the predicted value, and the "actual reward signal in the empirical samples + expected value of the next state" as the target value, the deviation between the two is calculated. The smaller the deviation, the more accurate the prediction of the value network. The loss function adopts the mean squared error form, and the overall loss value is obtained by calculating the average of the squared differences between the predicted values ​​and the target values ​​of all extracted samples. This overall loss value comprehensively reflects the prediction accuracy of the value network.

[0121] Step S154: Optimize the parameters of the value network structure of the agent structure through the backpropagation algorithm, adjust the weights and biases of the hidden layers of the value network structure of the agent structure, reduce the loss function value, until the prediction accuracy of the value network structure of the agent structure reaches a preset threshold.

[0122] The backpropagation algorithm is initiated, propagating the loss function value backward along the layers of the value network to calculate the contribution (gradient) of each layer's weights and biases to the loss value. Based on the gradient direction, the weight matrix and bias vector of the hidden layers are adjusted according to a preset learning rate to reduce the loss function value. For example, if the gradient of a certain layer's weight is positive, the weight value is decreased; if the gradient is negative, the weight value is increased.

[0123] The process of sample extraction, loss calculation, and parameter adjustment is repeated. After each round of adjustment, the prediction accuracy of the value network is tested using a validation sample set (separated from the experience replay buffer). When the prediction accuracy reaches a preset threshold (e.g., the deviation between the predicted and actual values ​​is less than a certain range), the value network optimization is stopped.

[0124] Step S155: Based on the parameters of the optimized value network structure of the agent structure, update the policy network structure of the agent structure, adjust the probability distribution of action selection in the policy network structure of the agent structure, increase the probability of generating actions corresponding to qualified process adjustment schemes, and decrease the probability of generating actions corresponding to unqualified process adjustment schemes.

[0125] Since the policy network and value network share the weight parameters of the first hidden layer, the basic feature extraction capability of the policy network has been improved synchronously after the value network is optimized. Based on this, the parameters of the policy network's output layer are further adjusted: the types of adjustment actions corresponding to qualified solutions in the experience replay buffer are statistically analyzed, and the frequency of each action is calculated; the weights of the output layer are adjusted according to the frequency, increasing the probability value of qualified actions with high occurrence frequency and decreasing the probability value of unqualified actions with low occurrence frequency.

[0126] For example, if the action combination of "fine-tuning forging temperature + coordinated adjustment of cooling flow" appears most frequently in the qualified scheme, then the weight of this action combination in the output probability distribution of the policy network will be increased, so that the agent is more inclined to choose this action combination in similar states.

[0127] Step S156: Collect the target data of the new process adjustment scheme generated by the adjusted agent structure and executed in the process optimization environment. Based on the target data, update the state space value range of the process optimization environment and the weight coefficient of the reward function of the process optimization environment to make the process optimization environment more in line with the actual processing scenario. The target data includes the new state information after the execution of the new process adjustment scheme, the reward signal corresponding to the new process adjustment scheme, and the execution effect of the new process adjustment scheme.

[0128] The controlled agent interacts with the process optimization environment in multiple rounds to generate and execute new process adjustment plans, and collects target data during the execution process, including the adjusted status information (such as equipment status and quality indicators under the new process parameter combination), the corresponding reward signals, and the actual execution effect (such as whether the quality standard is met).

[0129] Based on these target data, the process optimization environment is updated: if it is found that the actual value of some state variables exceeds the original set range (such as the motor current being higher than the original upper limit under a new parameter combination), the value range of the corresponding variables in the state space is expanded; if it is found that the evaluation of certain adjustment actions by the reward function deviates significantly from the actual execution effect (such as the reward value of a certain effective adjustment being too low), the weight coefficient of the reward function is fine-tuned to make the feedback of the environment more in line with the actual processing effect.

[0130] Step S157: Control the updated agent structure to interact with the updated process optimization environment again to generate a new set of process adjustment schemes, and repeat the process of process effect evaluation, adjustment of the strategy parameters of the agent structure, and updating of the process optimization environment.

[0131] The updated interaction process between the agent and the environment is initiated, and a new set of process adjustment schemes is generated according to the logic of step S130. Then, the process effect is evaluated according to the process of step S140. If the proportion of qualified schemes in the evaluation results does not meet expectations, the parameter adjustment and environment update process of steps S151-S156 is repeated until the proportion of qualified schemes continues to increase and tends to stabilize.

[0132] Step S158: Set preset optimization targets. The preset optimization targets include the proportion of qualified process adjustment schemes reaching a preset ratio, the average improvement of quality indicators reaching a preset value, the average improvement of efficiency reaching a preset value, and the average reduction of cost reaching a preset value. When the process adjustment schemes generated by multiple consecutive interactions all meet the preset optimization targets, the process adjustment scheme with the best effect is selected.

[0133] The following sub-steps are used to set the target and select the optimal solution: Step S1581: Extract the benchmark values ​​of quality indicators, efficiency, and cost from the initial processing technology data set during the historical processing process. Combine the frequency of qualified process adjustment schemes during the historical process optimization process to determine the four component dimensions of the preset optimization target. The four component dimensions are the proportion of qualified process adjustment schemes, the average improvement of quality indicators, the average improvement of efficiency, and the average reduction of cost.

[0134] The benchmark values ​​for quality indicators are the average values ​​of each quality indicator (such as diameter accuracy and surface roughness) in the initial dataset; the benchmark value for efficiency is the average processing time per batch in the initial dataset; and the benchmark value for cost is the average processing cost per piece in the initial dataset. The benchmark for the percentage of qualified process adjustment schemes is the highest percentage of qualified schemes in historical optimizations. Based on this, the four dimensions of the preset optimization objectives are determined.

[0135] Step S1582: Calculate the preset value of the average improvement of quality indicators based on the benchmark values ​​of quality indicators in the initial processing technology data set. The preset value of the average improvement of quality indicators is a preset proportional multiple of the benchmark values ​​of quality indicators. Calculate the preset value of the average improvement of efficiency based on the benchmark values ​​of efficiency in the initial processing technology data set. The preset value of the average improvement of efficiency is a preset proportional multiple of the benchmark values ​​of efficiency. Calculate the preset value of the average reduction of cost based on the benchmark values ​​of cost in the initial processing technology data set. The preset value of the average reduction of cost is a preset proportional multiple of the benchmark values ​​of cost. Determine the preset proportion of qualified process adjustment schemes based on the frequency of occurrence of qualified process adjustment schemes during historical process optimization.

[0136] For example, the preset value for the average improvement of quality indicators is set as a certain percentage of the benchmark value of quality indicators (such as reducing the benchmark value by a certain percentage for accuracy deviation); the preset value for the average improvement of efficiency is set as a certain percentage of the benchmark value of efficiency (such as shortening the processing time by a certain percentage from the benchmark value); the preset value for the average reduction of cost is set as a certain percentage of the benchmark value of cost (such as reducing the cost by a certain percentage from the benchmark value); and the preset percentage for the proportion of qualified process adjustment schemes is set as a certain percentage (such as the proportion of qualified schemes being no less than a certain percentage).

[0137] Step S1583: After each interaction between the agent structure and the process optimization environment, extract the total number of process adjustment schemes generated in this interaction and the number of qualified process adjustment schemes generated in this interaction from the set of process adjustment schemes generated in this interaction and the corresponding process effect evaluation results. Calculate the percentage of qualified process adjustment schemes in this interaction. The percentage of qualified process adjustment schemes is equal to the number of qualified process adjustment schemes generated in this interaction divided by the total number of process adjustment schemes generated in this interaction.

[0138] For example, if the total number of solutions generated in a certain interaction is several, and the number of qualified solutions is several, then the proportion of qualified process adjustment solutions is the ratio of the two. If this ratio reaches the preset proportion, then this dimension meets the standard.

[0139] Step S1584: From the process effect evaluation results corresponding to all process adjustment schemes generated in this interaction, extract the quality index change value, efficiency change value, and cost change value of each process adjustment scheme. Calculate the average improvement of quality index in this interaction. The average improvement of quality index is equal to the sum of the quality index change values ​​of all process adjustment schemes divided by the total number of process adjustment schemes generated in this interaction. Using the same calculation method, calculate the average efficiency improvement and average cost reduction of this interaction respectively.

[0140] The average improvement in quality indicators is obtained by summing the quality indicator changes (positive values ​​indicate improvement) of all solutions in this interaction and dividing by the total number of solutions. The average improvement in efficiency and cost is calculated using the same logic. If the calculation result reaches the preset value, the corresponding dimension meets the standard.

[0141] Step S1585: Compare the percentage of qualified process adjustment schemes calculated this time with the preset percentage of qualified process adjustment schemes, compare the average improvement of quality indicators calculated this time with the preset value of the average improvement of quality indicators, compare the average improvement of efficiency calculated this time with the preset value of the average improvement of efficiency, and compare the average reduction of cost calculated this time with the preset value of the average reduction of cost.

[0142] Compare the calculation results of the four dimensions with the preset values ​​one by one. If all four dimensions meet the preset requirements (such as the proportion meeting the standard, the quality improvement meeting the standard, the efficiency improvement meeting the standard, and the cost reduction meeting the standard), then it is determined that the solution generated by this interaction meets the preset optimization goal as a whole.

[0143] Step S1586: If the proportion of qualified process adjustment schemes obtained in this calculation is greater than or equal to the preset proportion of qualified process adjustment schemes, the average improvement of quality indicators obtained in this calculation is greater than or equal to the preset value of the average improvement of quality indicators, the average improvement of efficiency obtained in this calculation is greater than or equal to the preset value of the average improvement of efficiency, and the average reduction of cost obtained in this calculation is greater than or equal to the preset value of the average reduction of cost, then mark the process adjustment scheme generated in this interaction as meeting the preset optimization target.

[0144] The interaction batches that meet all the requirements are marked, and the set of solutions and evaluation results of the batch are recorded to prepare for the subsequent selection of the optimal solution.

[0145] Step S1587: Record the number of interactions that meet the preset optimization target using a counter. If the process adjustment scheme generated by this interaction meets the preset optimization target, the counter value is incremented by 1; if the process adjustment scheme generated by this interaction does not meet the preset optimization target, the counter value is reset to 0.

[0146] The counter is set to an initial value of 0. When an interaction meets the target, the counter is incremented by 1. If subsequent interactions do not meet the target, the counter is reset to 0, ensuring that only the number of consecutive interactions that meet the target is counted.

[0147] Step S1588: When the counter value reaches the preset number of times threshold, extract the process adjustment scheme generated by each interaction within the preset number of times threshold to form a set of process adjustment schemes to be screened.

[0148] If the preset threshold for the number of interactions is set to a certain value, when the counter reaches that value, it indicates that the solutions generated by the consecutive interactions have met the optimization objective. At this point, all solutions from these interactions are extracted to form a set to be filtered.

[0149] Step S1589: Extract the quality dimension influence coefficient, efficiency dimension influence coefficient, and cost dimension influence coefficient from the initial processing technology data set. The sum of the quality dimension influence coefficient, efficiency dimension influence coefficient, and cost dimension influence coefficient is 1.

[0150] Based on the principle of prioritizing quality in crankshaft machining, the quality dimension influence coefficient is set to the maximum, while the efficiency and cost dimension influence coefficients are set according to the company's production needs. The sum of the three is 1, which is used to calculate the comprehensive evaluation score of the scheme.

[0151] Step S15810: For each process adjustment scheme in the set of process adjustment schemes to be screened, extract the improvement of quality indicators, the improvement of efficiency, and the reduction of cost from the corresponding process effect evaluation results, and combine them with the influence coefficients of the quality dimension, the efficiency dimension, and the cost dimension to calculate the independent score of the process adjustment scheme in each dimension; finally, determine the comprehensive evaluation score based on the combination results of the independent scores.

[0152] The quality score is obtained by multiplying the quality improvement of each solution by the quality dimension influence coefficient, the efficiency score is obtained by multiplying the efficiency improvement by the efficiency dimension influence coefficient, and the cost score is obtained by multiplying the cost reduction by the cost dimension influence coefficient. The three are added together to obtain the comprehensive evaluation score.

[0153] Step S15811: Sort the comprehensive evaluation scores of all process adjustment schemes and select the process adjustment scheme with the highest comprehensive evaluation score as the best process adjustment scheme.

[0154] The solutions are ranked from highest to lowest based on their comprehensive evaluation scores, with the solution having the highest score being the one that performs best in terms of quality, efficiency, and cost.

[0155] Step S159: Refine the optimal process adjustment plan, determine the specific process parameter data for each process, the specific operating status requirements for each piece of equipment, and the specific execution sequence of each process, and form and output the final automotive parts processing optimization plan.

[0156] For example, the following sub-steps can be used to improve and output the solution: Step S1591: Extract the process parameter data, equipment operating status parameter data and process sequence data from the process adjustment solution with the best effect, separate the data of each type and establish a data index.

[0157] The optimal solution is analyzed, and the process parameters (such as temperature, pressure, speed, and feed rate) of each process, such as forging, turning, milling, and grinding, as well as the operating status parameters (such as current, vibration frequency, and cooling flow rate) of each piece of equipment (forging equipment, lathe, and milling machine), and the execution order of each process are extracted. A unique index is established for each type of data to facilitate subsequent processing.

[0158] Step S1592: Input the extracted process parameter data into the parameter standardization module. According to the parameter receiving format of the processing equipment, convert the process parameter data into machine-recognizable numerical parameters. Specifically, the processing temperature parameter is converted into a temperature numerical format compatible with the equipment's temperature control system, the processing pressure parameter is converted into a pressure numerical format compatible with the equipment's pressure control unit, the tool speed parameter is converted into a speed numerical format compatible with the equipment's spindle control unit, and the processing feed parameter is converted into a feed numerical format compatible with the equipment's feed system.

[0159] To meet the parameter interface requirements of different equipment, process parameters are formatted: for example, forging temperature is converted into a numerical format that the temperature control system can recognize, ensuring that the temperature control unit can accurately receive and execute parameters; tool speed is converted into a format compatible with the spindle control unit to avoid equipment failure due to format mismatch.

[0160] Step S1593: Input the extracted equipment operating status parameter data into the equipment parameter mapping module, and associate the unique equipment identifier with the corresponding operating status parameters. Among them, the equipment motor operating current parameter corresponds to the current parameter specification of the equipment electrical control interface, the equipment transmission component vibration frequency parameter corresponds to the frequency parameter specification of the equipment vibration monitoring module, the equipment cooling system flow parameter corresponds to the flow parameter specification of the equipment cooling control unit, and the equipment component temperature parameter corresponds to the temperature parameter specification of the equipment temperature monitoring interface. Based on the standardized process parameter data and equipment operating status parameter data, a process parameter mapping table is generated through the data association module. The process parameter mapping table contains the correspondence between process identifier, equipment identifier, process parameter value, and equipment operating status parameter value.

[0161] Each processing equipment is assigned a unique identifier (e.g., "Forging Equipment-F01", "Lathe-C02"), and the operating status parameters are associated with the equipment identifier to ensure a one-to-one correspondence between parameters and equipment. The generated process parameter mapping table clearly records the equipment and parameter requirements corresponding to each process. For example, "Forging Process-01" corresponds to "Equipment-F01", the process parameters are "Temperature XXX, Pressure XXX", and the operating status parameters are "Current XXX, Cooling Flow Rate XXX".

[0162] Step S1594: Input the extracted process sequence data into the process logic verification module. According to the process dependency relationship of automotive parts processing, verify whether the sequential logic of each process in the process sequence data conforms to the processing flow. If there is a process logic conflict, trigger the sequence adjustment instruction to adjust the process sequence data to conform to the processing flow.

[0163] The process logic verification module verifies the rationality of the process sequence in the optimal solution based on the inherent process flow of crankshaft machining (such as forging → rough turning → finish turning → milling → grinding → inspection). For example, if the "grinding" process is placed before "finish turning" in the solution, a logical conflict is determined, and an adjustment command is automatically triggered to move "grinding" after "finish turning" to ensure that the process sequence meets the actual machining requirements.

[0164] Step S1595: Input the verified process sequence data and process parameter mapping table into the process document generation module. According to the preset structured document format, integrate the process sequence, process parameter values, equipment operating status parameter values ​​and equipment identification information to generate a structured process document containing data field identifiers and parameter values.

[0165] The structured process document adopts the enterprise standard format and includes a header (scheme name, version, preparation date), a process list (process number, name, adjustment content, corresponding equipment identifier), a process parameter table (process number, parameter type, parameter value, allowable deviation), an equipment status requirement table (equipment identifier, status parameter type, parameter value range), and remarks (applicable batches of the scheme, precautions), etc.

[0166] For example, the header section indicates "Crankshaft Machining Process Optimization Scheme V1.0, Compilation Date: XXX"; in the process list, "Process 01: Forging" corresponds to "Equipment Identifier F01," and the adjustment content is "Temperature Fine-tuning + Cooling Flow Optimization"; the parameter types for "Process 01" in the process parameter table include "Forging Temperature" and "Forging Pressure," clearly specifying the corresponding parameter values ​​and allowable deviations; the status parameters for "Equipment F01" in the equipment status requirements table include "Motor Current" and "Cooling Flow," indicating the parameter value range. Through structured integration, the document content is ensured to be clear and logically coherent, facilitating understanding and execution by production personnel.

[0167] Step S1596: Input the structured process document into the equipment control interface test module, simulate the processing equipment to receive parameter data in the structured process document, and detect the receiving response status of the equipment control interface to each parameter. If all parameters can be received normally by the equipment control interface, the test is considered to have passed. If there are parameters that cannot be received, return to the parameter standardization module for reprocessing.

[0168] The equipment control interface testing module establishes a simulated testing environment, including the same control interface protocols (such as Modbus and Profinet) as the actual processing equipment (forging equipment F01, lathe C02, etc.). Process parameters and equipment status parameters from the structured process document are encapsulated into test commands according to the interface protocol format and sent one by one to the simulated control interface of the corresponding equipment.

[0169] Monitor the interface's receive response status: If the parameter data format meets the interface requirements, the simulated device returns a "received successfully" response; if the parameter data format does not match (e.g., incorrect value type, parameter range exceeds limits), the simulated device returns a "received failed" response and indicates the error type (e.g., "parameter value exceeds device rated range"). For parameters that fail to be received, return to the parameter standardization module to readjust the parameter format or value range until all parameters can be normally received by the simulated interface, and the test passes.

[0170] Step S1597: After the test is passed, the structured process document is input into the format conversion module, converted into a process document format compatible with the enterprise production management system, and stored in the process scheme database of the production management system.

[0171] Enterprise production management systems typically store process plans using specific document formats (such as XML, JSON, or custom formats). The format conversion module converts the content of structured process documents into the corresponding format according to the system interface specifications. During the conversion process, the correspondence between data fields remains unchanged, ensuring that core information such as process parameters and equipment status requirements are completely preserved.

[0172] For example, parameters such as "forging temperature" and "forging pressure" and their corresponding values ​​are converted into tag fields that the production management system can recognize, and the data is organized according to the hierarchical structure required by the system. After the conversion is completed, the document is uploaded to the process scheme database through the system interface, and an index is created (such as by crankshaft model and processing batch) to facilitate subsequent retrieval and retrieval.

[0173] Step S1598: Through the interface of the production management system, send the converted process document to the control system of each processing equipment, and push the process document update notification to the production scheduling module to complete the output of the final automotive parts processing optimization plan.

[0174] The production management system sends the converted process documents to the corresponding control systems of forging equipment F01, lathe C02, etc., via industrial Ethernet. After receiving the documents, each equipment control system updates its local process parameter configuration to ensure that the equipment operates according to the optimized plan. At the same time, after receiving the update notification, the production scheduling module pushes a reminder message to the production scheduler (such as "Crankshaft machining process plan has been updated, applicable to batch XXX"), notifying the scheduler to arrange the production plan.

[0175] After the output is completed, the production management system records the output log, including information such as the output time, the list of receiving equipment, and the document version, to facilitate subsequent tracking of the implementation status of the solution. The final output process optimization solution can directly guide the actual crankshaft machining production, achieving the optimization goals of quality improvement, efficiency enhancement, and cost reduction.

[0176] Figure 2 shows a schematic diagram of the hardware structure of an automotive parts processing optimization system 100 based on reinforcement learning, provided in an embodiment of the present invention, for implementing the above-described method for optimizing automotive parts processing technology based on reinforcement learning. As shown in Figure 2, the automotive parts processing optimization system 100 based on reinforcement learning may include a processor 110, a machine-readable storage medium 120, a bus 130, and a communication unit 140.

[0177] In the specific implementation process, one or more processors 110 execute computer-executable instructions stored in machine-readable storage medium 120, so that processor 110 can execute the automotive parts processing optimization method combined with reinforcement learning as described in the above method embodiment. The processor 110, machine-readable storage medium 120 and communication unit 140 are connected through bus 130. The processor 110 can be used to control the sending and receiving actions of communication unit 140.

[0178] The specific implementation process of processor 110 can be found in the various method embodiments executed by the above-mentioned automotive parts processing technology optimization system 100 combined with reinforcement learning. The implementation principle and technical effect are similar, and will not be repeated here.

[0179] Furthermore, this embodiment of the invention also provides a readable storage medium containing computer-executable instructions. When the processor executes the computer-executable instructions, the above-mentioned method for optimizing the automotive parts processing technology in conjunction with reinforcement learning is implemented.

[0180] It should be noted that, in order to simplify the description of this invention and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of this invention sometimes combines multiple features into a single embodiment, drawing, or description thereof. Similarly, it should be noted that, in order to simplify the description of this invention and thus aid in the understanding of one or more embodiments, the foregoing description of the embodiments of this invention sometimes combines multiple features into a single embodiment, drawing, or description thereof.

Claims

1. A method for optimizing the processing technology of automotive parts by combining reinforcement learning, characterized in that, The method includes: Obtain an initial processing technology data set, which includes process parameter data, equipment operating status data, and product quality inspection data during the processing of automotive parts; Based on the initial processing technology data set, a reinforcement learning-based process optimization environment and agent structure are constructed. The process optimization environment is used to simulate the processing technology execution scenario, and the agent structure is used to generate process adjustment strategies. The intelligent agent structure is controlled to interact with the process optimization environment. The intelligent agent structure generates a process adjustment plan based on the state information fed back by the process optimization environment. The process optimization environment outputs a corresponding set of reward signals based on the process adjustment plan. The process adjustment schemes are combined with corresponding reward signals to evaluate the process effect, and it is determined whether the processing quality, efficiency and cost indicators corresponding to each process adjustment scheme meet the preset requirements, so as to obtain the process effect evaluation results. Based on the process effect evaluation results, adjust the strategy parameters of the intelligent agent structure, update the process optimization environment based on the adjusted intelligent agent structure, repeat the interaction and evaluation process until a process adjustment scheme that meets the preset optimization target is obtained, and output the final automotive parts processing process optimization scheme.

2. The method for optimizing automotive parts processing technology using reinforcement learning as described in claim 1, characterized in that, The acquisition of the initial processing technology data set includes: Collect process parameter data corresponding to each process in the processing of automotive parts. The process parameter data includes processing temperature data, processing pressure data, tool speed data, and processing feed data. Record the process parameter data when each process is executed. Collect equipment operating status data during the operation of the processing equipment. The equipment operating status data includes equipment motor operating current data, equipment transmission component vibration frequency data, equipment cooling system flow data, and equipment component temperature data. The equipment operating status data is acquired and stored in real time. Obtain the quality inspection data of the tested automotive parts, which includes the dimensional accuracy data, surface roughness data, mechanical performance data and defect detection data of the parts, and organize the product quality inspection data corresponding to each batch of products. The collected process parameter data, equipment operating status data, and product quality inspection data are classified and integrated, and the data are associated according to the order of processing steps to form a data association mapping relationship; Based on the data association mapping relationship, duplicate and invalid data entries are removed, and data with complete process correspondence is retained to generate an initial processing technology data set containing process parameter data, equipment operating status data, and product quality inspection data.

3. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 1, characterized in that, The process optimization environment and agent structure built based on the initial processing technology data set include: Extract key variables related to process execution from the initial processing technology data set, determine the state space dimension of the process optimization environment, the state space dimension includes process parameter variable dimension, equipment state variable dimension and quality index variable dimension, and define the value range of each state space dimension; Based on the adjustment requirements of automotive parts processing technology, the types of process adjustment actions are divided, including process parameter fine-tuning actions, equipment status calibration actions, and process sequence adjustment actions. A reward function for the process optimization environment is constructed. The input of the reward function is the standardized value of the change in quality indicators, efficiency, and cost after process adjustment. The reward value is calculated according to the preset weight coefficient. When quality is improved, efficiency is increased, and cost is reduced, the reward value is positive, and vice versa. A policy network structure is constructed for the intelligent agent structure. The policy network structure includes an input layer, a hidden layer, and an output layer. The input layer receives the state information of the process optimization environment. The hidden layer processes the state information through a fully connected layer and an activation function. The output layer outputs the probability distribution of each process adjustment action. Configure the value network structure of the agent structure. The value network structure is used to evaluate the expected value of each action in the current state. The input is state information and action information, and the output is the expected reward value. Integrate the policy network structure and the value network structure to form a complete agent structure. The state space dimension, action type, and reward function are associated with the agent structure to generate a process optimization environment and agent structure that simulates the execution scenario of the processing technology and supports the interaction of the agent structure through reinforcement learning.

4. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 3, characterized in that, The reward function for constructing the process optimization environment takes as input changes in quality indicators, efficiency, and cost after process adjustment. The reward value is calculated based on preset weighting coefficients. A positive reward value is generated when quality improves, efficiency increases, and cost decreases, and a negative reward value is generated otherwise. This includes: The input parameter type of the reward function is determined, and the input parameters are the quality index change value, efficiency change value, and cost change value after process adjustment. The quality index change value is the difference between the quality index after process adjustment and the quality index before process adjustment. The efficiency change value is the difference between the processing time before process adjustment and the processing time after process adjustment. The cost change value is the difference between the processing cost before process adjustment and the processing cost after process adjustment. The weighting coefficients for each input parameter are determined by expert experience. The weighting coefficients for the quality index change values ​​are set according to the degree of influence of the quality of automotive parts on the performance of automotive products. The weighting coefficients for the efficiency change values ​​are set according to the degree of influence of processing efficiency on the enterprise's production plan. The weighting coefficients for the cost change values ​​are set according to the degree of influence of processing costs on the enterprise's production efficiency. The sum of the weighting coefficients for the quality index change values, the efficiency change values, and the cost change values ​​is 1. The mathematical expression of the reward function is constructed, and the reward value is obtained by standardizing and adding the products of the quality indicator change value and the weight coefficient of the quality indicator change value, the efficiency change value and the weight coefficient of the efficiency change value, and the cost change value and the weight coefficient of the cost change value. The reward value is adjusted according to the following rules: when the change value of the quality indicator is positive, the change value of the efficiency is positive, and the change value of the cost is positive, the reward value is positive; when the change value of any input parameter is negative, the reward value is reduced accordingly based on the weight coefficient of the change value of that input parameter; if the change values ​​of two or three input parameters are negative, the reward value is negative. To verify the reward function, select a portion of historical process adjustment data from the initial processing technology dataset, input the changes in quality indicators, efficiency, and cost after the historical process adjustments into the reward function, calculate the reward value, and compare the calculated reward value with the actual effect of the historical process adjustments to determine whether the reward value can accurately reflect the actual effect of the historical process adjustments. Based on the verification results, adjust the weight coefficients of the changes in each input parameter. If the deviation between the reward value and the actual effect of the historical process adjustment is greater than the set deviation threshold, adjust the weight coefficients of the corresponding input parameter changes. Repeat the verification and adjustment process until the reward value can accurately reflect the comprehensive impact of the process adjustment on quality, efficiency and cost, and determine the final reward function.

5. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 1, characterized in that, The control system interacts with the intelligent agent structure and the process optimization environment. The intelligent agent structure generates a process adjustment plan based on the state information fed back by the process optimization environment. The process optimization environment outputs a corresponding reward signal based on the process adjustment plan, including: The process optimization environment is initialized by taking the initial process parameter data, initial equipment operating status data, and initial product quality inspection data in the initial processing process data set as the initial state information of the process optimization environment and sending the initial state information to the intelligent agent structure. The initial state information is received by the agent structure and input into the policy network structure of the agent structure. The initial state information is processed by the hidden layer of the policy network structure of the agent structure. The output layer of the policy network structure of the agent structure generates the probability distribution of each process adjustment action. The process adjustment action with the highest probability is selected according to the probability distribution and combined to form an initial process adjustment scheme. The initial process adjustment plan is input into the process optimization environment, which simulates the execution of the initial process adjustment plan, updates process parameter data and equipment operating status data, and calculates the corresponding product quality inspection data based on preset quality assessment rules. Based on the updated process parameter data, equipment operating status data, and product quality inspection data, the corresponding reward signal is calculated through the reward function of the process optimization environment. The reward signal reflects the impact of the initial process adjustment scheme on processing quality, efficiency, and cost. The calculated reward signal and updated state information are fed back to the intelligent agent structure through the process optimization environment. The intelligent agent structure receives the new state information and repeats the above process of generating a process adjustment scheme, executing the process adjustment scheme in the process optimization environment, and calculating the reward signal. Record the process adjustment schemes and corresponding reward signals generated in each interaction, and organize them into a set of process adjustment schemes and a set of corresponding reward signals according to the interaction order.

6. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 5, characterized in that, The process involves receiving the initial state information through the agent structure, inputting it into the policy network structure of the agent structure, processing the initial state information through the hidden layer of the policy network structure, generating a probability distribution of each process adjustment action through the output layer of the policy network structure, selecting the process adjustment action with the highest probability based on the probability distribution, and combining them to form an initial process adjustment scheme, including: The initial state information sent by the process optimization environment is received through the input layer of the policy network structure of the intelligent agent structure, and converted into a vector form that can be processed by the policy network structure of the intelligent agent structure. The initial state information includes initial process parameter data, initial equipment operating status data, and initial product quality inspection data. The transformed state vector is input into the hidden layer of the policy network structure of the agent structure. The hidden layer of the policy network structure of the agent structure contains multiple fully connected layers. The first fully connected layer of the hidden layer of the policy network structure of the agent structure performs a linear transformation on the state vector and maps the state vector to a higher-dimensional feature space through the weight matrix to obtain the first feature vector. The first feature vector is input into the activation function, which uses the ReLU function to suppress the negative values ​​in the first feature vector and retain the positive features in the first feature vector to generate the activated second feature vector. The second feature vector is input to the next fully connected layer below the hidden layer of the policy network structure of the agent structure, and the linear transformation and activation operations are repeated. After multiple layers of processing by the hidden layer of the policy network structure of the agent structure, the final high-level feature vector is input to the output layer of the policy network structure of the agent structure. The output layer of the policy network structure of the intelligent agent structure is a softmax layer. The high-level feature vectors are normalized, and the probability value corresponding to each process adjustment action is calculated. The sum of the probability values ​​of all process adjustment actions is 1, forming the probability distribution of each process adjustment action. Extract the probability value of each process adjustment action from the probability distribution of each process adjustment action, compare the probability of all process adjustment actions, and select the process adjustment action with the highest probability. If there are multiple process adjustment actions with the same and highest probability, then determine one of the process adjustment actions by random selection. Based on the selected process adjustment actions and the sequence of automotive parts processing, the adjustment range of process parameters, the adjustment method of equipment operating status, and the details of process execution corresponding to the process adjustment action are determined, and integrated to form an initial process adjustment plan.

7. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 1, characterized in that, The process adjustment scheme is combined with the corresponding reward signal to evaluate the process effect, and it is determined whether the processing quality, efficiency and cost indicators corresponding to each process adjustment scheme meet the preset requirements, so as to obtain the process effect evaluation result, including: Extract the process parameter adjustment content, equipment operation status adjustment content, and process adjustment content corresponding to each process adjustment scheme from the set of process adjustment schemes; Based on the reward signals in the reward signal set that correspond one-to-one with the process adjustment schemes, the quality index change value, efficiency change value, and cost change value corresponding to each process adjustment scheme are analyzed. The quality index change value corresponds to the improvement or decline in product quality, the efficiency change value corresponds to the shortening or extension of the processing cycle, and the cost change value corresponds to the reduction or increase in processing cost. Preset requirements for processing quality, efficiency, and cost are set. The preset requirements for processing quality include the allowable deviation range of dimensional accuracy, the allowable range of surface roughness, and the standard for mechanical properties. The preset requirements for efficiency include the upper limit of processing time per batch. The preset requirements for cost include the lower limit of processing cost per piece. The changes in quality indicators, efficiency, and cost corresponding to each process adjustment scheme are compared with the corresponding preset requirements to determine whether the quality indicators meet the allowable deviation range of dimensional accuracy, the allowable range of surface roughness, and the standard of mechanical properties; whether the efficiency meets the upper limit of single batch processing time; and whether the cost meets the lower limit of single piece processing cost. Count the number of indicators that meet the preset requirements in each process adjustment plan. If all three indicators of quality, efficiency and cost meet the preset requirements, the process adjustment plan is marked as a qualified process adjustment plan; if two indicators meet the preset requirements, the process adjustment plan is marked as a process adjustment plan to be improved; if less than two indicators meet the preset requirements, the process adjustment plan is marked as a unqualified process adjustment plan. The marking results of all process adjustment schemes are compiled, and combined with the changes in quality indicators, efficiency, and cost corresponding to each process adjustment scheme, a process effect evaluation result is formed, which includes the qualification status, indicator compliance status, and specific change values ​​of each process adjustment scheme.

8. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 1, characterized in that, The process involves adjusting the strategy parameters of the intelligent agent structure based on the evaluation results of the process effect, updating the process optimization environment based on the adjusted intelligent agent structure, repeating the interaction and evaluation process until a process adjustment scheme that meets the preset optimization target is obtained, and outputting the final automotive parts processing process optimization scheme, including: Qualified process adjustment schemes and corresponding reward signals are selected from the process effect evaluation results. The adjustment actions of the qualified process adjustment schemes, the status information before and after the execution of the qualified process adjustment schemes, and the reward signals corresponding to the qualified process adjustment schemes are combined into an experience sample and stored in the experience playback buffer. Multiple experience samples are randomly selected from the experience replay buffer. The state information of each experience sample is input into the value network structure of the agent structure to calculate the expected value of each action in the current state. At the same time, the next state information of each experience sample is input into the value network structure of the agent structure to calculate the expected value of the next state. Based on the reward signal, the expected value of the current state, and the expected value of the next state in each experience sample, a loss function is constructed. The loss function is used to measure the deviation between the predicted value of the value network structure of the agent structure and the actual reward signal and the value of the next state. The parameters of the value network structure of the agent structure are optimized by backpropagation algorithm, and the weights and biases of the hidden layer of the value network structure of the agent structure are adjusted to reduce the loss function value until the prediction accuracy of the value network structure of the agent structure reaches a preset threshold. Based on the parameters of the value network structure of the optimized agent structure, the policy network structure of the agent structure is updated, the probability distribution of action selection in the policy network structure of the agent structure is adjusted, the probability of generating the action corresponding to the qualified process adjustment scheme is increased, and the probability of generating the action corresponding to the unqualified process adjustment scheme is reduced. The target data of the new process adjustment scheme generated by the adjusted intelligent agent structure is collected and executed in the process optimization environment. Based on the target data, the state space value range of the process optimization environment and the weight coefficient of the reward function of the process optimization environment are updated to make the process optimization environment more in line with the actual processing scenario. The target data includes the new state information after the execution of the new process adjustment scheme, the reward signal corresponding to the new process adjustment scheme, and the execution effect of the new process adjustment scheme. The updated agent structure interacts with the updated process optimization environment again to generate a new set of process adjustment schemes. The process of evaluating process effects, adjusting the strategy parameters of the agent structure, and updating the process optimization environment is repeated. Set preset optimization goals, which include the percentage of qualified process adjustment schemes reaching a preset ratio, the average improvement of quality indicators reaching a preset value, the average improvement of efficiency reaching a preset value, and the average reduction of cost reaching a preset value. When the process adjustment schemes generated by multiple consecutive interactions all meet the preset optimization goals, the process adjustment scheme with the best effect is selected. The optimal process adjustment plan is refined in detail, the specific process parameters of each process, the specific operating status requirements of each piece of equipment, and the specific execution sequence of each process are determined, and the final optimized process plan for automotive parts processing is generated and output.

9. The method for optimizing automotive parts processing technology by combining reinforcement learning according to claim 8, characterized in that, The preset optimization goals include achieving a preset percentage for the proportion of qualified process adjustment schemes, a preset value for the average improvement in quality indicators, a preset value for the average improvement in efficiency, and a preset value for the average reduction in cost. When multiple consecutive process adjustment schemes generated through interaction all meet the preset optimization goals, the process adjustment scheme with the best effect is selected, including: The baseline values ​​of quality indicators, efficiency, and cost in the historical processing process are extracted from the initial processing technology data set. Combined with the frequency of qualified process adjustment schemes in the historical process optimization process, four component dimensions of the preset optimization target are determined. The four component dimensions are the proportion of qualified process adjustment schemes, the average improvement of quality indicators, the average improvement of efficiency, and the average reduction of cost. The preset values ​​for the average improvement of quality indicators are calculated based on the benchmark values ​​of quality indicators in the initial processing technology dataset. These preset values ​​are multiples of the benchmark values. Similarly, the preset values ​​for the average improvement of efficiency are calculated based on the benchmark values ​​of efficiency in the initial processing technology dataset. The preset values ​​for the average improvement of cost are also calculated based on the benchmark values ​​of cost in the initial processing technology dataset. Finally, the preset percentage of qualified process adjustment schemes is determined based on the frequency of qualified process adjustment schemes during historical process optimization. After each interaction between the intelligent agent structure and the process optimization environment, the total number of process adjustment schemes generated in this interaction and the number of qualified process adjustment schemes generated in this interaction are extracted from the set of process adjustment schemes generated in this interaction and the corresponding process effect evaluation results. The percentage of qualified process adjustment schemes in this interaction is calculated, which is equal to the number of qualified process adjustment schemes generated in this interaction divided by the total number of process adjustment schemes generated in this interaction. From the process effect evaluation results of all process adjustment schemes generated in this interaction, extract the quality index change value, efficiency change value, and cost change value of each process adjustment scheme. Calculate the average improvement of quality indicators in this interaction. The average improvement of quality indicators is equal to the sum of the quality index changes of all process adjustment schemes divided by the total number of process adjustment schemes generated in this interaction. Using the same calculation method, calculate the average efficiency improvement and average cost reduction of this interaction respectively. The percentage of qualified process adjustment schemes calculated this time is compared with the preset percentage of qualified process adjustment schemes. The average improvement of quality indicators calculated this time is compared with the preset value of the average improvement of quality indicators. The average improvement of efficiency calculated this time is compared with the preset value of the average improvement of efficiency. The average reduction of cost calculated this time is compared with the preset value of the average reduction of cost. If the proportion of qualified process adjustment schemes obtained in this calculation is greater than or equal to the preset proportion of qualified process adjustment schemes, the average improvement of quality indicators obtained in this calculation is greater than or equal to the preset value of the average improvement of quality indicators, the average improvement of efficiency obtained in this calculation is greater than or equal to the preset value of the average improvement of efficiency, and the average reduction of cost obtained in this calculation is greater than or equal to the preset value of the average reduction of cost, then the process adjustment scheme generated in this interaction is marked as meeting the preset optimization target. The counter records the number of interactions that meet the preset optimization goals. If the process adjustment plan generated by this interaction meets the preset optimization goals, the counter value is incremented by 1; if the process adjustment plan generated by this interaction does not meet the preset optimization goals, the counter value is reset to 0. When the counter value reaches the preset number of times threshold, the process adjustment scheme generated by each interaction within the preset number of times threshold is extracted to form a set of process adjustment schemes to be screened. The influence coefficients of quality, efficiency, and cost are extracted from the initial processing technology data set. The sum of the influence coefficients of quality, efficiency, and cost is 1. For each process adjustment scheme in the set of process adjustment schemes to be screened, the improvement in quality indicators, the improvement in efficiency, and the reduction in cost are extracted from the corresponding process effect evaluation results. These are then combined with the influence coefficients of the quality dimension, the efficiency dimension, and the cost dimension, respectively, to calculate the independent score of the process adjustment scheme in each dimension. Finally, the comprehensive evaluation score is determined based on the combination of the independent scores. The comprehensive evaluation scores of all process adjustment schemes are ranked, and the process adjustment scheme with the highest comprehensive evaluation score is selected as the best process adjustment scheme.

10. A system for optimizing the processing technology of automotive parts by combining reinforcement learning, characterized in that, The automotive parts processing technology optimization system combining reinforcement learning includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to run the programs, instructions or code in the memory to implement the automotive parts processing technology optimization method combining reinforcement learning as described in any one of claims 1-9.

Images