A knowledge-driven reinforcement learning process planning scheme generation method

Abstract

The application provides a kind of knowledge-driven reinforcement learning process planning scheme generation method, belongs to process planning technical field.The method comprises: the structure and content of process design requirement are identified, which is matched with specific instance in process knowledge graph;Process requirement characteristics and process scheme structure are represented as a unified adjacency matrix based on CDK model;Set the state space, action space and reward mechanism of deep reinforcement learning;Output the final state space, parse the text vector in the node table and label each relationship to obtain the design scheme.The application realizes the deep integration and ordered calling of case knowledge and rule knowledge by combining deep reinforcement learning with knowledge graph, can intelligently convert multi-element process design requirements into a complete process design scheme, can be uniformly applied to machining process, connection process and assembly process planning scenarios, effectively improves the utilization efficiency of process knowledge and the intelligent level of process design.

Description

Technical Field

[0001] This invention relates to the field of process planning technology, and in particular to a knowledge-driven reinforcement learning method for generating process planning schemes. Background Technology

[0002] As modern manufacturing accelerates its evolution towards automation, informatization, and intelligence, the processing tasks of complex products are becoming increasingly demanding, and the requirements for processing quality are continuously improving. This places higher demands on process design, necessitating its transformation towards intelligence to meet the high standards of efficiency and effectiveness required by intelligent manufacturing systems.

[0003] Traditional process planning relies heavily on the experience and knowledge of human experts. Due to the lack of automation and intelligent technologies, its decision-making efficiency is limited, and it struggles to guarantee the optimality of process solutions. In the evolution of intelligent process design, computers have largely replaced manual process planning methods. Computer-aided process planning (CAPP), as a bridge between CAD and CAM, can effectively improve the processing capabilities of complex processes, transform product design into the optimal manufacturing sequence, and rationally allocate manufacturing resources. However, traditional CAPP methods still face significant challenges in the flexible reuse of process knowledge and the generation of process solutions. Specifically, these challenges manifest as difficulties in knowledge accumulation, inflexible reuse mechanisms, and fragmented reasoning processes. These problems are not limited to machining processes but are also prevalent in joining processes (such as welding and gluing) and assembly processes (such as bolted connections and press fitting), necessitating the introduction of more advanced and intelligent technologies to overcome these challenges.

[0004] In recent years, artificial intelligence (AI) technology has been widely applied in the industrial manufacturing sector, demonstrating powerful capabilities in large-scale data processing and complex mapping modeling, injecting new momentum into the manufacturing industry's transition from automation to intelligence. To adapt to the development requirements of intelligent manufacturing, process design must also evolve towards intelligence. Currently, significant progress has been made in the intelligent design of machining, joining, and assembly processes, encompassing methods such as matching and reuse based on historical cases and local reasoning based on fragmented rules. However, in the comprehensive utilization of case knowledge and rule knowledge to support the intelligent generation of process solutions, problems remain, including insufficient flexibility in case reuse and fragmented rule reasoning, which hinder the unified organization and efficient reuse of process knowledge in the context of intelligence.

[0005] Deep reinforcement learning (DRL), with its inherent advantages in sequential decision-making and dynamic learning, offers a novel approach to addressing complex and dynamic process planning problems. Combining the structured interpretability of knowledge graphs (KG) with the environmental awareness capabilities of DRL provides a key path to solving these challenges. While numerous studies have attempted to apply deep reinforcement learning to manufacturing scenarios, research that systematically combines knowledge graphs with deep reinforcement learning for process design scheme generation remains scarce. Therefore, there is an urgent need to construct a generalized deep reinforcement learning-based framework for process design scheme generation to support the efficient utilization of process knowledge and intelligent scheme generation. Summary of the Invention

[0006] The purpose of this invention is to provide a knowledge-driven reinforcement learning method for generating process planning schemes, so as to obtain reference schemes that better meet the process design intentions, thereby continuously improving the proposed process requirements, and finally forming a new scheme that can guide actual production. After the generated scheme is verified, it can also be directly stored in the case knowledge base to achieve rapid knowledge accumulation.

[0007] To achieve the above objectives, this invention provides a knowledge-driven reinforcement learning-based method for generating process planning schemes, comprising the following steps: Step S1: Identify the structure and content of the process design requirements, match the process design requirements with specific instances in the process knowledge graph, and obtain an instantiated representation of the initial process requirements. Step S2: Based on the CDK model, the instantiated initial process requirements are mapped to an initial scheme space. The initial scheme space uses an adjacency matrix to represent the process requirement characteristics and process scheme structure. Step S3: Using the initial scheme space as the initial state space for deep reinforcement learning, set the state representation method and state update rules for the state space; Step S4: Set up the action space for deep reinforcement learning, and use the actions in the action space to gradually update the state space; Step S5: Set up a reward mechanism for deep reinforcement learning and calculate the reward value based on the update result of the state space after the action space is executed; Step S6: Output the final state space after action update, parse the text vectors in the node table, and label each relationship to obtain the design scheme.

[0008] Therefore, the present invention employs the above-mentioned knowledge-driven reinforcement learning method for generating process planning schemes, and the beneficial technical effects are as follows: (1) Fully identify the structure and content of the diverse process design requirements and match them with specific instances in the process knowledge graph; on this basis, allocate the process elements related to the process requirements to the corresponding space based on the CDK definition and merge them to obtain the adjacency matrix that represents the requirements characteristics and the scheme structure.

[0009] (2) Select appropriate rule knowledge, take into account the integrity of case knowledge nodes, and filter suitable rules when existing cases cannot meet the requirements, so as to achieve the organic combination of case and rule knowledge; then set up state space, action space and reward mechanism to build a deep reinforcement learning model.

[0010] (3) Output the final state space, parse the text vectors in the node table and label each relation to obtain the design scheme, thereby transforming the process design requirements into a reference scheme that can guide production, making it easier for designers to improve the requirements and form new schemes. Attached Figure Description

[0011] Figure 1 This is a flowchart of a knowledge-driven reinforcement learning method for generating process planning schemes according to the present invention. Figure 2 This is a schematic diagram of the overall framework of a knowledge-driven reinforcement learning-based process planning scheme generation method according to the present invention. Figure 3 This is a schematic diagram of the structure of the present invention, which extracts process requirement nodes and maps them to the solution space. Figure 4 This is a schematic diagram of the deep reinforcement learning model algorithm for generating process solutions according to the present invention; Figure 5 This invention provides a comparative analysis of the DDPG-based framework with other typical DRL frameworks; wherein, Figure 5 In the diagram, (a) shows the learning curves for different DRL frameworks. Figure 5 (b) in the figure is the F-test heatmap of the DRL framework; Figure 6 This is an example of generating a machining solution for the present invention. Detailed Implementation

[0012] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0013] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0014] Example 1 See Figure 1 and Figure 2 A knowledge-driven reinforcement learning method for generating process planning schemes, comprising: Step S1: Identify the structure and content of the process design requirements and match them with specific instances in the process knowledge graph to obtain an instantiated representation of the initial process requirements.

[0015] This embodiment uses three steps—top-level knowledge modeling, case and rule knowledge unit construction, and concept-decision-knowledge space representation—to unify and structure the case and rule knowledge with varying structures in the process. The example used is machining.

[0016] See Figure 3 Based on the top-level knowledge model of machining processes, and using natural language processing technologies such as word segmentation and named entity recognition, process nodes composed of tags and attributes are extracted from the design requirement text of process engineers to form a requirement list. To match entities in the knowledge graph, the tags and attributes in the nodes are combined into short texts (such as...). The semantic similarity between the tag (name, attribute, value) and the knowledge graph node is calculated using text representation tools such as text2vec. Matching results with a similarity greater than 0.80 are taken as the instantiation representation of the initial process requirements.

[0017] The requirements for machining process design take many forms, as shown in Table 1. After requirement identification, the machining process design requirement entities can be obtained as shown in Table 2.

[0018] Table 1 Machining process design requirements

[0019] Table 2 Entities of Machining Process Design Requirements

[0020] By calculating semantic similarity, instance nodes such as parts, processes, equipment, and features are matched to process requirements from the knowledge graph. Instantiated representations of the state space are constructed using nodes with high similarity, resulting in initial states under various label combinations. Utilizing the knowledge graph to supplement unclear process information in the requirements not only helps designers clarify the requirements but also provides a more comprehensive initial state space for the deep reinforcement learning model. Furthermore, the multiple state spaces generated by various instantiation representations help ensure that the generated solution covers as many possible scenarios as possible.

[0021] Step S2: Based on the Concept-Decision-Knowledge (CDK) model, the instantiated initial process requirements are mapped to an initial solution space. The initial solution space uses an adjacency matrix to represent the process requirement characteristics and process solution structure.

[0022] Based on the definition of process CDK space representation The space, as the process requirement space, represents a collection of diverse design requirements such as parts, processes, and features. By mapping process requirements into a requirement matrix containing a node table and an adjacency matrix, a requirement space that conforms to the top-level knowledge model specification is formed. Space as the primary decision-making space, based on For the instantiation requirements of the space, the key main concepts in the smallest subgraph that can cover the initial requirements are selected, and the process backbone information closely related to the requirements is generated through the decision matrix. Space as a supplementary resource space includes Other entity concepts outside of space are supplemented by the resource matrix to provide process information that was not explicitly mentioned in the initial requirements but was specified in the top-level model.

[0023] Based on this, the initial process requirements represented by the instantiation are incorporated into the process requirement adjacency matrix to obtain the adjacency matrix. Meanwhile, the process of generating the process scheme is abstracted into a process of gradually adding instantiated nodes and relationships to the adjacency matrix, so it is necessary to... The elements are allocated to key information and supplementary resources representing the process scheme. , Within the space, an initial scheme space representation is obtained by recombination to facilitate the generation of reinforcement learning schemes.

[0024] To transform the demand space into a solution space, it is necessary to... The concept of space is assigned to , In space.

[0025] Based on the instantiated process requirements, the smallest subgraph that can cover the proposed requirements concepts is selected under the top-level knowledge model specification, and the concepts contained within it are assigned to... Spatial union is represented as an adjacency matrix Concepts that were not included were assigned to Spatial union is represented as an adjacency matrix The specific process is as follows: The requirement tags are selected in pairs using a permutation and combination method, and the shortest undirected knowledge chain with them as the beginning and end is matched in the knowledge model. By merging all matching knowledge chains and removing duplicate labels, we obtain the minimum knowledge subgraph that covers the initial requirements. Its matrix dimension is The adjacency list is arranged with instantiated tags first and uninstantiated tags last. The remaining uninstantiated process tags are obtained as supplementary resources. The matrix dimension is ; Will and Merge into an adjacency matrix , denoted as the initial scheme space The top left corner The area is Space, bottom right corner The area is Space is set to 0 for the remaining areas; This only includes the initial instantiation requirement; other entities and relationships need to be further instantiated and generated by the reinforcement learning model and used as model input for training and testing.

[0026] Step S3: Using the initial scheme space as the initial state space for deep reinforcement learning, set the state representation method and state update rules for the state space.

[0027] The state space represents the structural and content states of a process design scheme. The structure is defined by an uninstantiated top-level knowledge model, representing the process elements and their relationships within the scheme. The content is defined by case and rule knowledge, acquired and progressively improved from the knowledge graph by a reinforcement learning model during scheme generation. For ease of computation, both structure and content are represented as a state matrix. The adjacency matrix and node list represent the scheme structure and content, respectively. The dimension of the adjacency matrix is ​​the square of the sum of the number of nodes in the top-level knowledge model of the process plus 1, i.e. ,in Let be the dimension of the state space. This represents the total number of nodes in the process knowledge model.

[0028] Since the process scheme is generated starting from design requirements, the initial state space... The initial solution space obtained from requirement identification The scheme generation, or state space update process, involves instantiating a node in the node table to represent adding a new process entity, and changing any 0 to 1 in the adjacency matrix to represent creating a new process relationship. The final output is the updated state space. This constitutes a complete process design scheme.

[0029] Step S4: Set up the action space for deep reinforcement learning, and use the actions in the action space to gradually update the state space.

[0030] The task of the action space is to update the structure and content of the process plan using stored case and rule knowledge. The process of adding process cases and rule knowledge units of the <head entity, relation, tail entity> triple structure to the state space is transformed into the operation of selecting instances of the head entity or tail entity in the node table and creating relations at off-diagonal positions in the adjacency matrix. For ease of calculation, each action is represented as an action matrix. It uses the vector representations of the head and tail entities and the label position as key information for executing actions, thus forming... The simplified matrix reduces the dimension of the action space significantly without affecting the action matrix. For the state matrix Perform a state update. Each action space... The steps to be performed are as follows: Step a, set This is the time step in which the current action is executed. ; Step b: Based on the initial state space Randomly select an instantiated process entity from the node table as the head entity. Let its index position be . ; Step c: Establish new relationships for the first time, using the initial state space. middle Starting from the position, at a random position in the same row (not the starting point), change a value of 0 to 1, and establish a sequence from... The relationship of departure The position index of the resulting new relation is denoted as ; Step d, according to The index will be the index position in the node table. The process entity as the tail entity Determine the tail entity Whether to instantiate; if not instantiated, i.e., the value is 0, then the entity vector of the case or rule knowledge is randomly matched from the process knowledge graph; otherwise, the vector representation of the original instantiated process knowledge is maintained. Step e: From the above steps, we can obtain a triplet where both the head and tail entities are instantiated and a relationship is established between them. We also obtain the action matrix in the node table and adjacency matrix corresponding to the indexes of each element in the triplet. ,calculate Different positions are used The values ​​are replaced to obtain the updated state space. ; Step f: Configure the action space for each execution. Stored in action memory space In the middle, it is used to determine whether an action is repeated; Step g, when At that time, step b, establishing a new relationship, requires action memory space. Based on this, do not create duplicate relationships at locations where relationships have already been established; Step h, set This is the next time step after the action is completed. Repeat the above steps until the termination condition is met, i.e. .

[0031] Step S5: Set up a reward mechanism for deep reinforcement learning and calculate the reward value based on the update result of the state space after the action space is executed.

[0032] Set up a reward mechanism for generating process solutions, consisting of regional weights, unit rewards, and result evaluation. Figure 4 As shown in the flowchart, the meaning and significance of each part are as follows: Regional weights: according to (Key Entities) (The main body and supplementary resources are related) (Supplementary Resources) Weights are set for the three regions respectively. , , ,satisfy The weighting relationship in the initial reward Based on this, each reward is calculated using a weighted average: the reward value after regional weighting. This allows the model to prioritize the areas most relevant to the design requirements, among which This represents the weight of the corresponding region.

[0033] Unit Rewards: Define the reward direction and reward value for the process of obtaining rewards for matching cases and rule unit knowledge.

[0034] First, the reward direction is determined based on the new process knowledge added after the state update: let the reward direction coefficient be set. If the newly added knowledge exists in the knowledge graph and belongs to the case knowledge unit ( ), denoted as And the value is greater than 0; if it belongs to a rule knowledge unit ( ), denoted as And the value is greater than 0; if it does not exist, it is denoted as And the value is less than 0. The relationship between the three is as follows: This reflects a generation model that prioritizes case knowledge and supplements it with rule knowledge.

[0035] Then, the knowledge triple vector representation of the relation is used in TransE. , can be obtained abbreviated as Based on the knowledge units in the knowledge graph that share the same head and tail tags as newly added knowledge, it is represented as... Calculate the cosine similarity between the newly added knowledge and each knowledge unit with the same label. , recorded as The average of these values ​​represents the reward. When the newly added knowledge is a case study and a perfect match, the reward is... The mean is the ideal value of 1, where ( () represents the head entity vector of the newly added triple. ( ) represents the relation vector of the newly added triple. ( ) represents the tail entity vector of the newly added triple. This represents the head entity vector of a reference unit in a knowledge graph. Represents the relation vector of a reference unit in a knowledge graph. This represents the tail entity vector of the reference unit in the knowledge graph.

[0036] Combining regional weights and reward directions, the reward definition for each instance is as follows: ; in, The relation vectors of newly added knowledge and the first knowledge in the knowledge graph Cosine similarity of relation vectors with the same label and reference knowledge The average value of the similarity reward is taken as the reference to the total number of knowledge items.

[0037] Result evaluation: The state space at each step adjacency matrix The adjacency matrix corresponding to the top-level knowledge model Perform cosine similarity calculation, to As a scaling factor for the reward at the current time step, As a scaling factor for the reward at the next time step, the reward value will be amplified when the similarity between the state matrix and the top-level model increases, and conversely, the reward value will be reduced when the similarity decreases.

[0038] ; ; Reward scaling factor ; In conclusion, The reward value for each action is obtained by updating the state, and then the total reward value is obtained by accumulating these values.

[0039] To further validate the model's performance, taking a machining process planning scenario as an example, the framework based on Deep Deterministic Policy Gradient (DDPG) is compared with other typical Deep Reinforcement Learning (DRL) frameworks, including Actor-Critic (A2C), SoftActor-Critic (SAC), and Proximal Policy Optimization (PPO). Each framework is trained 10 times based on the same machining process knowledge graph, with the cumulative reward every 200 steps used as the round reward. The reward change curves for 500 rounds are recorded, and the mean and variance of the 10 training rounds are calculated. An F-test is then performed on the reward results of each framework. Experimental results are as follows: Figure 5 As shown in Table 3, compared with frameworks based on A2C, SAC, and PPO, the DDPG-based framework has a faster reward convergence speed and achieves a higher reward value with similar training time. Specifically, the DDPG-based framework achieves a best reward of 72.393, significantly higher than A2C (50.824), SAC (42.855), and PPO (12.765). Furthermore, at a significance level of 0.05, Figure 5 The q-value heatmap results further demonstrate that the DDPG-based framework outperforms other comparative frameworks, with q-values ​​of 0.000, significantly lower than 0.05. Therefore, the DDPG-based framework exhibits superior performance in the machining process design generation task of this method.

[0040] Table 3 Training time and learning results of different DRL frameworks

[0041] Compared to the traditional process design process that relies on human experience, the method proposed in this embodiment can effectively reduce the cost of designers repeatedly adjusting process schemes. By leveraging the ability of the DRL method to extract and internalize process knowledge from diverse requirements, it can uncover the implicit relationships of process knowledge and apply them efficiently to the design process.

[0042] Step S6: Output the final state space after action update, parse the text vectors in the node table, and label each relationship to obtain the design scheme.

[0043] Taking the machining requirements of blind holes in cylinder head parts as an example, the required machining process design scheme is generated through model inference. Leveraging the interpretability advantages of reinforcement learning, this method can be better integrated with actual business operations. Based on the partitioning of the scheme space and the setting of the reward mechanism, the generation process is mainly divided into two parts: key information generation and supplementary resource generation. In the key information generation part, the model prioritizes areas with higher reward weights. Spatial elements are generated. For example, elements not mentioned in the requirements but belonging to... In the top-level model, the process steps are key nodes connecting processes and features. Therefore, process-related cases or rule units matched from the knowledge graph receive greater reward weight. Combining the evaluation of the overall solution with each generation, the model prioritizes finding process knowledge with the highest unit reward, thus avoiding reduced overall rewards or increased penalties due to later generation. In the supplementary resource generation section, the model generates K-space elements such as tools and auxiliary tools that are not mentioned in the requirements later, in order to generate a solution with a higher overall reward and a final result that better meets the machining process design requirements.

[0044] Furthermore, the generated machining process design scheme will have different content depending on the instantiated content, which process engineers can quickly understand from the final process scheme. For example... Figure 6 As shown, for the design requirement of "when performing datum surface machining on a cylinder head part on a machining center with the code TH6563×63A, how should the blind hole feature be machined? Please provide some reference process solutions," the tool can be a GB / T4256 Φ16 reamer, and the selected tool corresponds to the basic dimension of the blind hole feature Φ16±0.1 in the corresponding instantiated requirement. In addition to providing tool information that meets the process design, some generated solutions start from the example requirement, match the initial requirement, and additionally provide information on the dedicated fixture corresponding to the machining center. Based on the fuzzy blind hole requirement, it provides multiple feasible steps such as drilling, boring, and reaming. This method can efficiently obtain macroscopic solution information such as parts, processes, steps, equipment, fixtures, and tools, providing a solution for process engineers to quickly acquire effective knowledge from complex and diverse machining process knowledge.

[0045] This method is also applicable to generating solutions for connection processes (such as welding, gluing, and riveting). Unlike machining processes, the requirements identification for connection processes focuses on factors such as base material, joint type, connection strength requirements, and working environment. When constructing the adjacency matrix based on the CDK model, The space will include nodes such as "base material", "joint type", and "connection method"; The space focuses on key decision-makers such as "welding methods (e.g., MIG, TIG)," "adhesive types," and "rivet specifications." The space is supplemented with process parameters such as "preheating temperature," "protective gas flow rate," "curing time," and "riveting pressure." The definition logic of the state space and action space is completely consistent with that of the machining process, but the node table and relation types need to be replaced with knowledge graph entities specific to the connection process. Regarding the reward mechanism, since the connection process is more sensitive to process parameters, the weight of rule knowledge in the unit reward calculation can be appropriately increased so that the model can more flexibly combine parameter rules to generate connection schemes that meet strength and quality requirements. Apart from this, the similarity evaluation between the state matrix and the top-level model, the training of the DDPG model, and the scheme output process are all the same as in the machining process implementation example.

[0046] The application of this method in assembly process planning (such as bolted connections, press fitting, and snap-fit ​​assembly) maintains the core process consistent with machining processes, but the connotations of process elements differ. The requirements identification phase needs to extract key information such as "assemblies," "datum parts," "assembly sequences," and "fit tolerances." In CDK space mapping, The space describes the set of parts to be assembled and their spatial constraints. The space is responsible for generating key decisions such as "assembly sequence", "positioning method", and "fastening strategy"; The decision space stores supplementary process details such as "tightening torque," "pressing speed," and "adhesive application specifications." The execution logic of the action space remains unchanged, but each action corresponds to selecting an assembly operation or relationship in the assembly knowledge graph. A significant difference compared to machining processes lies in the decision space of the assembly process. With stronger sequential dependencies, the reward mechanism incorporates a penalty for the feasibility of the assembly sequence during the outcome evaluation phase. For example, if a certain action prevents the installation of subsequent critical parts, the reward for that step is significantly reduced through a scaling factor. This design difference allows the model to prioritize logical assemblability when generating assembly solutions. The final output assembly solution, in the form of a knowledge graph, can be directly used to guide the sequencing of assembly operations and resource allocation on the production line.

[0047] This method represents case knowledge and rule knowledge as a unified triple, and sets the unit reward for case knowledge to be greater than that for rule knowledge. This allows the model to prioritize the use of structurally complete case knowledge when rewards are similar, and then consider structurally fragmented rule knowledge. This balances the advantages of case knowledge in terms of solution completeness with the flexibility of rule knowledge in local reasoning, making the generated solutions more reliable and flexible in meeting actual process design business and needs.

[0048] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0049] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0050] Therefore, this invention adopts the above-mentioned knowledge-driven reinforcement learning method for generating process planning schemes. Based on CDK space partitioning and deep reinforcement learning framework, it realizes the deep integration and orderly calling of case knowledge and rule knowledge. It can intelligently transform diverse process design requirements into structurally complete and logically reasonable process design schemes, which can be uniformly applied to various planning scenarios such as machining processes, connection processes and assembly processes. It improves the utilization efficiency of process knowledge and the level of intelligence of process design. The verified schemes can be stored in the case knowledge base to realize the rapid accumulation and continuous reuse of knowledge.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating process planning schemes using knowledge-driven reinforcement learning, characterized in that, Includes the following steps: Step S1: Identify the structure and content of the process design requirements, match the process design requirements with specific instances in the process knowledge graph, and obtain an instantiated representation of the initial process requirements. Step S2: Based on the CDK model, the instantiated initial process requirements are mapped to an initial scheme space. The initial scheme space uses an adjacency matrix to represent the process requirement characteristics and process scheme structure. Step S3: Using the initial scheme space as the initial state space for deep reinforcement learning, set the state representation method and state update rules for the state space; Step S4: Set up the action space for deep reinforcement learning, and use the actions in the action space to gradually update the state space; Step S5: Set up a reward mechanism for deep reinforcement learning and calculate the reward value based on the update result of the state space after the action space is executed; Step S6: Output the final state space after action update, parse the text vectors in the node table, and label each relationship to obtain the design scheme.

2. The knowledge-driven reinforcement learning-based process planning scheme generation method according to claim 1, characterized in that, In step S1, the structure and content of the process design requirements are identified, and the process design requirements are matched with specific instances in the process knowledge graph to obtain an instantiated representation of the initial process requirements, specifically including: Based on the top-level knowledge model of the process, relevant process concepts are extracted from the design requirement text submitted by process engineers, forming a requirement list composed of tags and attributes, transforming the process text into the format of process nodes; and then the nodes are... The included tags and attribute information are combined into short text. A text representation tool is used to calculate the similarity between the process requirement node and the knowledge graph node. Matching results with a similarity higher than 0.80 are taken as the instantiation representation of the initial process requirement.

3. The knowledge-driven reinforcement learning-based process planning scheme generation method according to claim 1, characterized in that, In step S2, the initial process requirements represented by the instantiation are mapped to the initial scheme space based on the CDK model, specifically including: The initial process requirements, represented by instantiation, are incorporated into the process requirement adjacency matrix to obtain the adjacency matrix. ; By selecting requirement labels pairwise using a permutation and combination method, the shortest undirected knowledge chain with these labels as the start and end points is matched in the knowledge model. All matched knowledge chains are then merged, and duplicate labels are removed to obtain the minimum knowledge subgraph covering the initial requirements. The adjacency list is arranged with instantiated labels first and uninstantiated labels last; the remaining uninstantiated process labels are used as supplementary resources to obtain the adjacency matrix. ;Will and Merge into an adjacency matrix , denoted as the initial scheme space .

4. The knowledge-driven reinforcement learning-based process planning scheme generation method according to claim 3, characterized in that, In step S3, the initial scheme space is used as the initial state space for deep reinforcement learning. The state representation method and state update rules of the state space are set, specifically including: The structure and content of the scheme are represented as a state matrix. The structure and content of the scheme are represented by an adjacency matrix and a node table, respectively. The dimension of the adjacency matrix is ​​the square of the sum of the number of nodes in the top-level knowledge model of the process plus 1. Initial scheme space As the initial state space for deep reinforcement learning, the scheme generation is the state space update process: each instantiation of a node in the node table represents the addition of a new process entity, and each 0 in the adjacency matrix becomes 1, representing the creation of a new process relationship.

5. The knowledge-driven reinforcement learning-based process planning scheme generation method according to claim 4, characterized in that, In step S4, the action space for deep reinforcement learning is set up, and the state space is updated step by step using the actions in the action space. Specifically, this includes: Each action is represented as an action matrix. It uses the vector representations of the head and tail entities and the label position as key information for executing actions, thus forming... A simplified matrix of dimension 1.

6. The knowledge-driven reinforcement learning-based process planning scheme generation method according to claim 5, characterized in that, The specific steps for updating the state space using the action space each time are as follows: Step a, set This is the time step in which the current action is executed. ; Step b: Based on the initial state space Randomly select an instantiated process entity from the node table as the head entity. Let its index position be . ; Step c, with the initial state space middle Starting from the position, at a random position in the same row (not the starting point), change a value of 0 to 1, and establish a sequence from... The relationship of departure The position index of the resulting new relation is denoted as ; Step d, according to The index will be the index position in the node table. The process entity as the tail entity Determine the tail entity Whether to instantiate; if not instantiated, randomly match the entity vector of case or rule knowledge from the process knowledge graph; otherwise, maintain the original vector representation of instantiated process knowledge. Step e: Obtain triples where both the head and tail entities are instantiated and a relationship is established between them; obtain the node table, adjacency matrix, and action matrix corresponding to the indexes of each element in the triples. ,calculate Different positions are used The values ​​are replaced to obtain the updated state space. ; Step f: Configure the action space for each execution. Stored in action memory space middle; Step g, when At that time, step b, establishing a new relationship, requires action memory space. Based on this, do not create duplicate relationships at locations where relationships have already been established; Step h, set This is the next time step after the action is completed. Repeat the above steps until the termination condition is met.

7. The knowledge-driven reinforcement learning-based process planning scheme generation method according to claim 6, characterized in that, In step S5, a reward mechanism for deep reinforcement learning is set up, and the reward value is calculated based on the update result of the state space after the action space is executed. Specifically, this includes: Divide the state space into , , Region and give satisfaction The weights; Positive and negative rewards are set based on whether the newly added knowledge after each execution of the action space matches the case or rule knowledge, to satisfy the requirement. The directional relationship is used to calculate the reward value based on cosine similarity. The scaling factor is determined by the similarity between the state matrix and the top-level model, and the reward for each step is obtained and the total reward is accumulated. in, , , They are respectively D area, DK area, K The weighting coefficient of the region; , , These are the knowledge units for newly added knowledge matching cases, knowledge units for matching rules, and the reward direction coefficient when no knowledge is matched.

8. The knowledge-driven reinforcement learning method for generating process planning schemes according to claim 7, characterized in that, Determining the scaling factor based on the similarity between the state matrix and the top-level model specifically includes: Calculate the current state space adjacency matrix The adjacency matrix corresponding to the top-level knowledge model cosine similarity and the updated state space adjacency matrix and cosine similarity ; according to and The size relationship determines the scaling factor : 。 9. The knowledge-driven reinforcement learning method for generating process planning schemes according to claim 8, characterized in that, The calculation method for each reward is as follows: ； in, For the first The base reward value before the step is executed; For regional weighting coefficients, Take 1, 2, and 3, which correspond to respectively , , Three areas; For the first The action matrix to be executed step by step; The relation vectors of newly added knowledge and the first knowledge in the knowledge graph Cosine similarity of relation vectors with the same label and reference knowledge For reference to the total number of knowledge points, This is a case study knowledge unit. For rule-based knowledge units; The reward value updated after each action is performed. Multiply The final reward value is obtained, and then the total reward value is accumulated.

10. The knowledge-driven reinforcement learning method for generating process planning schemes according to claim 4, characterized in that, In step S6, the output state space is updated by actions to form the final state space. The text vectors in the node table are parsed, and each relation is labeled to obtain the design scheme, which specifically includes: The final state space obtained after the model's round-ending output is progressively updated through the action space. The final state space contains a node table and an adjacency matrix. The text vectors in the node table are parsed into specific process entities in the process knowledge graph. The entity associations are established by combining the adjacency matrix and each relationship is labeled, ultimately generating a process solution in the form of a knowledge graph.

Images