Production scheduling policy screening method based on reinforcement learning

Abstract

The application relates to a production scheduling strategy screening method based on reinforcement learning, which comprises the following steps: after obtaining a plurality of initial scheduling strategies, determining a state vector corresponding to production state information generated according to current disturbance event information and workshop environment information; performing reinforcement learning scoring on the plurality of initial scheduling strategies based on a preset multi-layer perception (MLP) and the state vector, and obtaining a probability distribution of the initial scheduling strategies according to the scoring, and selecting a plurality of candidate scheduling strategies from the plurality of initial scheduling strategies based on the probability distribution; determining performance semantic quantitative scoring of the candidate scheduling strategies according to a preset performance semantic reward function; and selecting candidate scheduling strategies with top K performance semantic quantitative scoring from the candidate scheduling strategies to obtain a screening result comprising the top K candidate scheduling strategies, wherein K is a positive integer not greater than the number of the candidate scheduling strategies.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and production scheduling decision technology, and in particular to a production scheduling strategy selection method based on reinforcement learning. Background Technology

[0002] In the industrial manufacturing sector, production scheduling is a core element in ensuring efficient workshop operation and on-time order delivery. With the increasing complexity of the production environment, disruptive events (such as sudden equipment failures, emergency order insertions, and material supply delays) occur frequently. These dynamic disturbances severely disrupt the original production plan, making production scheduling decisions face significant uncertainties and timeliness challenges.

[0003] To address scheduling needs in disruptive environments, related technologies typically employ semantic reasoning based on knowledge graphs or historical rule matching to generate scheduling strategies. Specifically, these technologies analyze current workshop disturbance events and equipment status information, utilizing pre-defined rule bases or historical case libraries to retrieve and recommend scheduling schemes that meet business constraints. However, these methods have significant inherent limitations: knowledge graph-based reasoning and rule scoring rely heavily on static entity associations and historical execution records for comprehensive evaluation. In actual production, the merits of candidate scheduling strategies are highly scenario-dependent. Even if a strategy satisfies semantic and rule constraints, its actual performance is influenced by multiple real-time dynamic factors such as current workshop equipment load rate, material inventory levels, personnel configuration, and the scope of disturbance propagation. The same scheduling strategy might perform well under one production state but, under another resource load state, not only fail to alleviate disturbances but could also trigger cascading delays. The static rule scoring mechanisms in these technologies cannot perceive and quantify these dynamic environmental variables, making it difficult to accurately predict the actual performance of candidate strategies after execution.

[0004] Furthermore, given the dynamic and changing production environment, it is impossible to rely on a large number of manually labeled scheduling samples to train complex scheduling models within a limited time window. Therefore, how to adaptively evaluate the expected performance of each strategy in a limited candidate strategy space, based on the current actual operating status of the workshop and historical feedback information, and select the optimal scheduling scheme for the current scenario, is a key technical problem that urgently needs to be solved in the field of production scheduling, given the lack of a large amount of manually labeled data.

[0005] Currently, no effective solution has been proposed to address the problem that production scheduling strategies based on static rules in related technologies struggle to achieve optimal decision-making under disturbed environments. Summary of the Invention

[0006] This application provides a production scheduling strategy selection method based on reinforcement learning, which at least solves the problem in related technologies that production scheduling strategies generated based on static rules are difficult to achieve optimal decision-making under disturbed environments.

[0007] This application provides a reinforcement learning-based production scheduling strategy selection method, comprising: after acquiring multiple initial scheduling strategies, determining a state vector corresponding to production state information generated based on current disturbance event information and workshop environment information, wherein the production state information is used to characterize the production scheduling operation state under the influence of the disturbance event; performing reinforcement learning scoring on the multiple initial scheduling strategies based on a preset multilayer perceptron (MLP) and the state vector, obtaining a probability distribution of the initial scheduling strategies based on the scoring, and selecting multiple candidate scheduling strategies from the multiple initial scheduling strategies based on the probability distribution; determining the performance semantic quantitative score of the candidate scheduling strategies according to a preset performance semantic reward function, wherein the performance semantic reward function is constructed based on a scheduling performance reward function and a semantic reward function, the scheduling performance reward function is used to calculate the expected performance parameters corresponding to the candidate scheduling strategies, and the semantic reward function is used to calculate the semantic matching degree between the state vector and the candidate scheduling strategies; selecting the candidate scheduling strategies ranked in the top K positions by the performance semantic quantitative score from the candidate scheduling strategies, obtaining a selection result including the top K candidate scheduling strategies, wherein K is a positive integer not greater than the number of candidate scheduling strategies.

[0008] Compared to related technologies, the production scheduling strategy selection method based on reinforcement learning provided in this application involves: after acquiring multiple initial scheduling strategies, determining a state vector corresponding to production state information generated based on current disturbance event information and workshop environment information, wherein the production state information is used to characterize the production scheduling operation state under the influence of disturbance events; performing reinforcement learning scoring on multiple initial scheduling strategies based on a preset multilayer perceptron (MLP) and the state vector, obtaining the probability distribution of the initial scheduling strategies based on the scores, and selecting multiple candidate scheduling strategies from the multiple initial scheduling strategies based on the probability distribution; determining the performance semantic quantitative score of the candidate scheduling strategies according to a preset performance semantic reward function, wherein the performance semantic reward function is constructed based on a scheduling performance reward function and a semantic reward function, the scheduling performance reward function is used to calculate the expected performance parameters corresponding to the candidate scheduling strategies, and the semantic reward function is used to calculate the semantic matching degree between the state vector and the candidate scheduling strategies; and selecting the top K candidate scheduling strategies from the candidate scheduling strategies based on the performance semantic quantitative score. A scheduling strategy is selected, resulting in a selection of the top K candidate scheduling strategies, where K is a positive integer not greater than the number of candidate scheduling strategies. A state vector containing multi-dimensional attributes such as equipment operation, production tasks, material supply, personnel configuration, order constraints, and disturbance events is constructed by combining the production state information corresponding to the current disturbance event information and workshop environment information. This state vector is then concatenated with the strategy feature vectors corresponding to each initial scheduling strategy and input into a multilayer perceptron (MLP) strategy network. Candidate scheduling strategies are selected based on the selection probability distribution output by the strategy network. A composite quantitative score is then applied to the candidate scheduling strategies using a performance-semantic reward function composed of a scheduling performance reward function and a semantic reward function. Finally, the top K ranked strategies are selected as the final selection results. This approach solves the problem in related technologies where production scheduling strategies generated based on static rules struggle to achieve optimal decision-making under disturbance environments. It enables adaptive evaluation and optimal strategy selection within a limited candidate strategy space based on real-time dynamic operating conditions and historical feedback information, even in the absence of a large number of manually labeled samples. This improves the dynamic response capability and execution performance of scheduling decisions under disturbance production environments.

[0009] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1This is a hardware structure block diagram of the terminal of the production scheduling strategy screening method based on reinforcement learning according to an embodiment of this application; Figure 2 This is a flowchart of a production scheduling strategy selection method based on reinforcement learning according to an embodiment of this application; Figure 3 This is a structural block diagram of a production scheduling strategy screening device based on reinforcement learning according to an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0012] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0013] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "a," "an," "an," "the," and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms "comprising," "including," "having," and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. "Multiple stages" used in this application refers to two or more stages. "And / or" describes the relationship between related objects, indicating that three relationships may exist; for example, "A and / or B" can represent: A alone, A and B simultaneously, and B alone. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0014] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal for the production scheduling strategy selection method based on reinforcement learning, according to an embodiment of this application. Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0015] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the reinforcement learning-based production scheduling strategy screening method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0016] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0017] This embodiment provides a reinforcement learning-based production scheduling strategy selection method running on the aforementioned terminal. Figure 2 This is a flowchart of a production scheduling strategy selection method based on reinforcement learning according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: After obtaining multiple initial scheduling strategies, determine the state vector corresponding to the production state information generated based on the current disturbance event information and workshop environment information. The production state information is used to characterize the production scheduling operation state under the influence of the disturbance event.

[0018] In this embodiment, the executing entity of the production scheduling strategy filtering method of this application includes any one of the control node, edge computing gateway, or industrial cloud platform running in the production scheduling management system.

[0019] In this embodiment, when the current disturbance event is triggered, the production status information collected on the industrial fieldbus is read, and multi-dimensional indicator data such as the current equipment operating load, task progress, material inventory level, personnel configuration and work-in-progress constraints in the workshop are parsed out. At the same time, the semantic embedding vector corresponding to the current disturbance event is obtained from the preset production scheduling knowledge graph. The numerical indicators are normalized and scaled and then concatenated with the semantic embedding vector in a fixed-dimensional order to generate a high-dimensional state vector that represents the global operating status of the workshop under the current disturbance environment.

[0020] It should be noted that after generating the initial scheduling strategy based on the production scheduling knowledge graph, the first scheduling strategy is only good in terms of semantic coherence, but its actual execution effect cannot be guaranteed. Therefore, the initial scheduling strategy is obtained as a candidate set, and reinforcement learning is used to find the strategy with strong executability in this candidate set.

[0021] Step S202: Based on the preset multilayer perceptron (MLP) and state vector, reinforcement learning scores are performed on multiple initial scheduling strategies, and the probability distribution of the initial scheduling strategies is obtained based on the scores. Based on the probability distribution, multiple candidate scheduling strategies are selected from the multiple initial scheduling strategies.

[0022] In this embodiment, the policy feature vectors corresponding to each policy in the initial scheduling policy set are horizontally concatenated with the generated high-dimensional state vectors to form multiple concatenated feature vectors. Each concatenated feature vector is input into a pre-trained multilayer perceptron (MLP) policy network. Through the output layer of the policy network, a real score representing the tendency of the policy to be selected in the current state is output for each initial scheduling policy. After normalization by the Softmax function, the probability distribution of all initial scheduling policies in the current workshop state is generated. The initial scheduling policies are sorted from high to low according to their probability values, and several policies with probability values ​​exceeding a preset threshold or ranking at the top are selected as the candidate scheduling policy set for the next round of evaluation.

[0023] Step S203: Determine the performance semantic quantitative score of the candidate scheduling strategy according to the preset performance semantic reward function. The performance semantic reward function is constructed based on the scheduling performance reward function and the semantic reward function. The scheduling performance reward function is used to calculate the expected performance parameters corresponding to the candidate scheduling strategy, and the semantic reward function is used to calculate the semantic matching degree between the state vector and the candidate scheduling strategy.

[0024] In this embodiment, a dual scoring calculation is performed for each policy in the candidate scheduling policy set: The first step involves calling the scheduling performance reward function to read the total delay time, average completion time, and on-time delivery rate that may be generated after the strategy is executed, and then calculating the performance reward value based on the actual execution effect through weighted summation. The second step involves calling the semantic reward function to perform bilinear mapping and dot product operations on the current high-dimensional state vector and the feature vector of the candidate policy, and calculating the semantic matching score between the two to characterize the semantic rationality and consistency of the policy in the current operating environment. The performance reward value and the semantic matching score are weighted and summed according to preset weights to obtain the comprehensive performance semantic quantitative score of each candidate scheduling strategy in the current scenario.

[0025] Step S204: Select the candidate scheduling strategies ranked in the top K by performance semantic quantization score from the candidate scheduling strategies to obtain the screening results including the top K candidate scheduling strategies, where K is a positive integer not greater than the number of candidate scheduling strategies.

[0026] In this embodiment, the comprehensive performance semantic quantitative scores of all strategies in the candidate scheduling strategy set are sorted in descending order; according to the preset number of recommended strategies K (e.g., K=5 or K=10), the top K strategies are selected as the optimal strategy screening results, and the screening results are pushed to the production control system for direct execution, or submitted to the scheduling personnel terminal for review and confirmation.

[0027] Through steps S201 to S204 above, after acquiring multiple initial scheduling strategies, the state vector corresponding to the production state information generated based on the current disturbance event information and workshop environment information is determined; based on a preset multilayer perceptron (MLP) and state vectors, reinforcement learning scoring is performed on the multiple initial scheduling strategies, and the probability distribution of the initial scheduling strategies is obtained based on the scores, and multiple candidate scheduling strategies are selected from the multiple initial scheduling strategies based on the probability distribution; the performance semantic quantitative score of the candidate scheduling strategies is determined according to a preset performance semantic reward function; from the candidate scheduling strategies, the candidate scheduling strategies ranked in the top K positions of the performance semantic quantitative score are selected to obtain the screening result including the top K candidate scheduling strategies. By constructing a multi-dimensional state vector from the production state information corresponding to the current disturbance event information and workshop environment information, and concatenating this state vector with the policy feature vectors corresponding to each initial scheduling strategy, the input is then fed into a multilayer perceptron (MLP) policy network. Candidate scheduling strategies are selected based on the selection probability distribution output by the policy network. Then, a composite quantitative score is performed on the candidate scheduling strategies using a performance-semantic reward function composed of a scheduling performance reward function and a semantic reward function. Finally, the top K ranked strategies are selected as the screening results. This solves the problem in related technologies where production scheduling strategies generated based on static rules are difficult to achieve optimal decision-making under disturbance environments. It enables adaptive evaluation and optimal strategy selection of a limited candidate strategy space based on real-time dynamic working conditions and historical feedback information, even in the absence of a large number of manually labeled samples. This improves the dynamic response capability and execution performance of scheduling decisions under disturbance production environments.

[0028] It should be noted that, in this embodiment, by constructing the production status information generated from the current disturbance event information and workshop environment information into a state vector composed of multi-dimensional vectors such as equipment operating load, task progress, material inventory, personnel configuration, order constraints, and disturbance events, a precise numerical representation of the global operating status of the production system under disturbance conditions is achieved. This provides a high-dimensional feature input that fits the actual working conditions for the dynamic evaluation and optimization of subsequent scheduling strategies. By concatenating the state vector with the strategy feature vectors corresponding to each initial scheduling strategy, and inputting the concatenated feature vector into a multilayer perceptron (MLP) strategy network, the probability distribution of each strategy being selected is output. Based on the probability distribution, a candidate strategy set is extracted. This achieves the ability to autonomously learn the matching relationship between the state and the strategy using historical feedback signals without requiring a large amount of manually labeled data. This quickly narrows the search range in the limited initial strategy space, solving the inefficiency problem caused by the reliance on manual rule selection in traditional methods. A performance-semantic reward function is constructed by weighting and combining a scheduling performance reward function and a semantic reward function. On the one hand, the scheduling performance reward function is used to calculate the expected performance parameters after the strategy is executed, quantifying and adapting in real time to the impact of dynamic load and material changes in the workshop on the actual execution effect of the strategy. On the other hand, the semantic reward function is used to calculate the semantic matching degree through a bilinear mapping between the state vector and the feature vector of the candidate strategy, ensuring that the selected strategies not only have excellent performance but also conform to the semantic logic constraints of the knowledge graph. This avoids the drawback of simply pursuing local performance and selecting semantically unreasonable strategies, achieving synergistic optimization of performance priority and semantic rationality. At the same time, by sorting the candidate scheduling strategies in descending order of the comprehensive performance semantic quantitative score and selecting the top K optimal strategies as the final selection results, a simple, direct and quantifiable strategy selection output scheme is provided for production scheduling decisions under disturbance conditions. This significantly enhances the adaptive decision-making ability and actual execution performance of the production scheduling system in response to sudden disturbance events under dynamic operating conditions.

[0029] In some embodiments, reinforcement learning scoring is performed on multiple initial scheduling policies based on a preset multilayer perceptron (MLP) and state vectors, through the following steps: Step 21: Determine the scheduling policy semantic vector corresponding to each initial scheduling policy, and concatenate all scheduling policy semantic vectors with the state vector to obtain the fused feature vector corresponding to each initial scheduling policy.

[0030] In this embodiment, the input of the Multilayer Perceptron (MLP) is the concatenation of the state vector and the scheduling policy semantic vector, i.e., the fused feature vector, using MLP[s,a] i ] represents, where s represents the state vector corresponding to the current production state, a i Let [s,a] represent the semantic vector of the i-th initial scheduling policy. iThe symbol ] represents the concatenation of the state vector and the scheduling policy semantic vector, z i The interactive feature score represents the fused feature vector output by the multilayer perceptron (MLP).

[0031] Step 22: Using a multilayer perceptron (MLP), learn the interaction features of the fused feature vectors, and score the interaction features according to the scoring function corresponding to the MLP to obtain a matching score. The scoring function is constructed based on the network parameters obtained by reinforcement learning based on a preset activation function. The network parameters are used to represent the weight matrix and bias vector of all layers in the multilayer perceptron.

[0032] In this embodiment, the scoring function is expressed as: , is a parameterized function composed of a multilayer perceptron (MLP).

[0033] In this embodiment, during reinforcement learning training, after each selected and executed scheduling policy, a reward value is calculated based on the actual scheduling performance after the policy execution, and the network parameters of the multilayer perceptron (MLP) are updated using the policy gradient algorithm. The policy gradient formula is expressed as: Where J(θ) is the objective function, R(s,a) represents the reward value obtained after choosing scheduling policy a in the current state s, θ represents the network parameters of the multilayer perceptron (MLP), and ∇ θ This indicates that the gradient of the network parameter θ is calculated.

[0034] In this embodiment, after sufficient training, the network parameters θ are fixed, encoding the experience of which scheduling strategy can achieve high scheduling performance under different states; in the inference stage, the trained multilayer perceptron (MLP) is used to directly select the scheduling strategy with the highest score from multiple initial scheduling strategies as the optimal scheduling strategy, given the current state vector.

[0035] Through steps 21 and 22 above, the state vector and the semantic vector of the candidate scheduling policy are concatenated and input into the multilayer perceptron (MLP) for interactive feature learning and scoring. The network parameters are optimized by using reinforcement learning reward feedback, so that the MLP can learn the matching relationship between the state and the policy, thereby achieving adaptive optimization of the candidate scheduling policy.

[0036] In some embodiments, the probability distribution of the initial scheduling policy is obtained based on the score, and multiple candidate scheduling policies are selected from multiple initial scheduling policies based on the probability distribution, which is achieved through the following steps: Step 31: Use the normalized exponential function to process the matching score and obtain the probability distribution corresponding to all initial scheduling strategies.

[0037] In this embodiment, the scores (matching scores) of all candidate strategies are normalized using the Softmax function to obtain the probability distribution of the candidate strategies: Where N represents the total number of initial scheduling policies, z i This represents the matching score of the i-th initial scheduling policy. This indicates that the i-th scheduling strategy a is selected under the current production scheduling operation state s. i The matching probability.

[0038] Step 32: In the probability distribution, determine the matching probability corresponding to each initial scheduling strategy, where the matching probability is used to characterize the probability of selecting an initial scheduling strategy in the production scheduling operation state.

[0039] In this embodiment, the matching probability value of each initial scheduling policy is between 0 and 1, and the sum of the matching probabilities of all policies is equal to 1. A policy with a higher matching probability indicates a stronger tendency to be selected as the final scheduling scheme in the current state.

[0040] Step 33: Based on the matching probability, select initial scheduling strategies with a matching probability greater than a preset probability threshold from multiple initial scheduling strategies to obtain multiple candidate scheduling strategies.

[0041] In this embodiment, the preset probability threshold is set according to the number of candidate strategies required in the actual scheduling scenario. When the matching probability is higher than the threshold, the corresponding initial scheduling strategy is retained as a candidate scheduling strategy. When the matching probability is lower than or equal to the threshold, the corresponding initial scheduling strategy is filtered out. Through threshold filtering, the initial scheduling strategy set is reduced, and strategies with high matching probabilities are retained for subsequent processing.

[0042] Through steps 31 to 33 above, the matching score is transformed into a unified probability distribution metric, so that the selection tendency of each initial scheduling strategy is expressed in probability form, and the initial scheduling strategy set is screened and reduced based on the probability threshold, thereby reducing the computational burden of subsequent processing while retaining high-quality candidate strategies.

[0043] In some embodiments, the performance semantic quantification score of the candidate scheduling strategy is determined according to a preset performance semantic reward function, which is achieved through the following steps: Step 41: Use the scheduling performance reward function to determine the expected performance parameter value corresponding to the candidate scheduling strategy, and use the semantic reward function to determine the semantic reward parameter value corresponding to the candidate scheduling strategy.

[0044] In this embodiment, the scheduling performance reward function is expressed as: R perf (a i )=-w1·Tardiness(a i)-w2·Makespan(a i )+ w3·OnTimeRate(a i ), where a i Let Tardiness(a) represent the i-th candidate scheduling strategy. i ) represents the total delay time after implementing this strategy. Makespan(a i OnTimeRate(a) represents the average completion time after implementing this strategy. i ) represents the on-time delivery rate after implementing the strategy; w1, w2, and w3 are weighting coefficients that satisfy w1+w2+w3=1. They are manually set according to business objectives to reflect the priority of different scheduling objectives; total delay time and average completion time are cost-type indicators, and the smaller their values, the better the strategy. On-time delivery rate is a benefit-type indicator, and the larger its value, the better the strategy.

[0045] In this embodiment, the semantic reward function is represented as: R sem (s,a i )=σ(s T Wa i ), where s represents the state vector corresponding to the current production scheduling operation state, a i Let W be the semantic vector of the i-th candidate scheduling policy, W be the learnable parameter matrix, and σ(·) be the Sigmoid function, which maps the result to the interval 0 to 1. This function has a bilinear form s T Wa i It captures the complex interactions between the state vector and the policy semantic vector across all dimensions. This is equivalent to projecting the state and policy onto the same comparable space through linear mapping and then calculating the dot product. The resulting semantic reward parameter value is used to characterize the degree of semantic matching between the candidate scheduling policy and the current production scheduling operation state.

[0046] Step 42: Weight the expected performance parameter value and the semantic reward parameter value according to the performance semantic reward function to obtain the performance semantic quantitative score corresponding to the candidate scheduling strategy.

[0047] In this embodiment, considering both scheduling performance and semantic matching information, the performance semantic reward function is defined as: R(a) i ,s)=αR perf (a i )+(1-α) R sem (s,a i ), where α∈[0,1] is a weight coefficient used to balance the weight ratio of scheduling performance reward and semantic reward; by adjusting the value of α, the performance semantic quantitative score can adaptively focus on the scheduling execution effect or semantic rationality in different scheduling scenarios, so as to achieve the collaborative optimization of optimal performance and semantic rationality.

[0048] Through steps 41 to 42 above, a two-dimensional quantitative evaluation of candidate scheduling strategies is achieved based on scheduling performance indicators and semantic matching degree. Continuous reward signals are used to reflect the performance differences between different strategies, enabling the reinforcement learning model to obtain a stable optimization direction during training and avoiding the selection of scheduling strategies that, although superior in performance, do not conform to knowledge logic.

[0049] In some embodiments, the expected performance parameters corresponding to candidate scheduling strategies are determined using a scheduling performance reward function, which is achieved through the following steps: Step 51: Obtain the state performance mapping parameter table associated with the scheduling performance reward function. The state performance mapping parameters include scheduling strategy, expected performance and the correspondence between the two. The state performance mapping parameter table is generated based on the historical scheduling strategy and the corresponding historical scheduling response performance data under the preset state. The scheduling strategy is associated with the corresponding production state information, and the historical scheduling response performance data is associated with the following target evaluation indicators: total delay time, average completion time and delivery rate.

[0050] In this embodiment, each record in the state performance mapping parameter table includes production state characteristics, scheduling strategy identifier, and multiple performance index values ​​after the strategy is executed in that state; historical scheduling response performance data comes from actual production scheduling execution records, and each historical record includes disturbance events, workshop state, selected strategy and its total delay time, average completion time, and on-time delivery rate after execution; by collecting a large number of historical scheduling samples, using production state and scheduling strategy as joint inputs and performance indexes after execution as supervision signals, the state performance mapping parameter table is trained and generated, so that the table can reflect the historical performance of each scheduling strategy under different production states.

[0051] Step 52: In the state performance mapping parameter table, detect the expected performance corresponding to the candidate scheduling strategy.

[0052] In this embodiment, candidate scheduling strategies and their corresponding production status information are used as search conditions. A matching query is performed in the state performance mapping parameter table to find historical records that match the strategy and the current production status. The performance index values ​​recorded in the record are extracted as expected performance. When there are multiple historical records in the state performance mapping parameter table that correspond to the same strategy and similar status, the average value of each performance index is taken as expected performance to reduce the evaluation error caused by fluctuations in a single record.

[0053] Step 53: Perform deviation standardization on the expected performance using the scheduling performance reward function to obtain the expected performance parameter values ​​corresponding to the candidate scheduling strategies.

[0054] In this embodiment, the deviation standardization process uses Min-Max normalization to map each expected performance indicator to the interval between 0 and 1, specifically as follows: Where x represents the raw value of a certain performance indicator, x min and x max These represent the minimum and maximum values ​​of the indicator in the state performance mapping parameter table, respectively. Total delay time and average completion time are cost-related indicators; the smaller the value, the better the strategy, and the smaller the normalized value, the better the performance. On-time delivery rate is a benefit-related indicator; the larger the value, the better the strategy, and the larger the normalized value, the better the performance. Deviation standardization eliminates the dimensional differences between different performance indicators, enabling weighted fusion of indicators under a unified scale.

[0055] Through steps 51 to 53 above, the expected performance of candidate strategies based on historical scheduling execution records is realized, and multi-dimensional performance indicators are converted into quantitative parameter values ​​of a unified scale through deviation standardization, providing standardized input for the weighted calculation of subsequent performance semantic quantitative scoring, so that scheduling performance data can be compared and integrated with semantic matching data in the same framework.

[0056] In some embodiments, the semantic reward parameter value corresponding to the candidate scheduling strategy is determined according to the semantic reward function, which is achieved through the following steps: Step 61: Perform word vector processing on the policy text of the candidate scheduling policy to obtain the first policy feature vector corresponding to the candidate scheduling policy.

[0057] In this embodiment, the policy text of the candidate scheduling strategy includes a description of scheduling actions, resource allocation instructions, and constraint setting information. The policy text is encoded by a pre-trained word vector model, mapping each word in the text to a low-dimensional dense vector. Then, a weighted average or pooling operation is performed on all word vectors to obtain a first policy feature vector of fixed dimensions. The first policy feature vector and the embedding vector of the corresponding policy entity in the knowledge graph are in the same semantic space, so that the semantic information of the policy text can be aligned and fused with the structured semantic information in the knowledge graph.

[0058] Step 62: Calculate the semantic matching degree corresponding to the first policy feature vector and the state vector, and process the semantic matching degree using a preset semantic reward function to generate a semantic reward value corresponding to the semantic matching degree. The semantic reward parameter value includes the semantic reward value, and the semantic reward function uses the Sigmoid function to perform semantic matching degree mapping transformation.

[0059] In this embodiment, the semantic matching degree is calculated using the dot product or cosine similarity between the first policy feature vector and the state vector, which measures the semantic consistency between the candidate scheduling policy and the current production scheduling operation state. The semantic reward function adopts a bilinear form to interactively model the state vector and the first policy feature vector, expressed as: R sem (s,a i )=σ(s T Wa i ), where s represents the state vector corresponding to the current production scheduling operation state, a i Let represent the first policy feature vector corresponding to the i-th candidate scheduling policy, W be the learnable parameter matrix used to project the state vector and policy feature vector to the same comparable space, and σ(·) be the Sigmoid function, which maps the semantic matching degree into a semantic reward value in the range of 0 to 1.

[0060] In this embodiment, the bilinear form s T Wa i It can capture the pairwise interaction relationships between the state vector and the policy feature vector in each dimension, and has a stronger expressive power than simple vector dot product; the learnable parameter matrix W is jointly optimized with the policy network during reinforcement learning training, so that the calculation of semantic matching degree can adaptively adjust the weight relationship between each dimension of the state and each dimension of the policy, thereby more accurately reflecting the differentiated evaluation criteria of the semantic rationality of the policy under different production states; the sigmoid function compresses the semantic matching degree to the interval between 0 and 1, so that the semantic reward value and the scheduling performance reward value are on the same order of magnitude, which is convenient for subsequent weighted fusion.

[0061] Through steps 61 to 62 above, the textual description information of candidate scheduling strategies is transformed into a computable semantic feature vector, and a semantic reward value is generated through bilinear interaction and Sigmoid mapping. This allows the semantic rationality of the strategy to be integrated into the reward feedback mechanism of reinforcement learning in a continuous quantification manner, while pursuing optimal scheduling performance and constraining the strategy to conform to the semantic logic of the knowledge graph.

[0062] In some embodiments, the following steps are performed before obtaining multiple initial scheduling policies: Step 71: Obtain the first scheduling strategy generated by reasoning based on the preset production scheduling knowledge graph, wherein the first scheduling strategy includes strategy text.

[0063] In this embodiment, the production scheduling knowledge graph includes five core entities and their semantic relationships: workshop status, disturbance events, scheduling intentions, scheduling decisions, and scheduling strategies. When a disturbance event is detected, the system uses the disturbance event node as the center, matches the corresponding scheduling intention from the knowledge graph through a semantic reasoning mechanism, and generates a scheduling strategy that conforms to business logic constraints based on the mapping relationship between intentions and decisions. Each strategy records the scheduling action, resource allocation scheme, and execution constraint information in the form of natural language text.

[0064] Step 72: Use the ConvE model to map the policy text into a second policy feature vector, and encode the policy semantic vector corresponding to the first scheduling policy based on the second policy feature vector.

[0065] In this embodiment, the ConvE model is a pre-trained knowledge graph representation learning model that maps all entities and relations in the knowledge graph into low-dimensional continuous vectors. For the policy text, the entity names and relation types involved are first extracted as semantic anchors, and then the embedding vectors of the corresponding entities are obtained from the ConvE model. The embedding vectors of each entity in the policy text are combined according to the policy semantic structure, and a fixed-dimensional policy semantic vector is obtained after vector concatenation or weighted averaging. This policy semantic vector is in the same semantic space as the embedding vectors of the corresponding policy entities in the knowledge graph, thereby ensuring the semantic consistency of subsequent similarity calculation and semantic matching.

[0066] Step 73: Based on the policy semantic vector, calculate the cosine similarity between any two first scheduling policies, and based on the cosine similarity, cluster all first scheduling policies to obtain multiple initial scheduling policy clusters. Each initial scheduling policy cluster includes at least two first scheduling policies with a cosine similarity greater than a set similarity threshold.

[0067] In this embodiment, the cosine similarity between any two first scheduling strategies is calculated by dividing the dot product of the semantic vectors of the strategies by the product of their moduli. Based on the cosine similarity between all pairs of strategies, hierarchical clustering or density-based clustering methods are used to cluster all first scheduling strategies, and strategies that are close in distance in the semantic space are grouped into the same cluster. A similarity threshold is set to control the degree of semantic consistency of strategies within a cluster. Strategies with a cosine similarity greater than the threshold are considered semantically similar strategies and are clustered into the same cluster. Strategies with a cosine similarity less than or equal to the threshold are assigned to different clusters or treated as outliers.

[0068] Step 74: Based on the semantic reward function, determine the semantic reward value of each first scheduling strategy in the initial scheduling strategy cluster, and select the first scheduling strategy with the highest semantic reward value from each initial scheduling strategy cluster to obtain the initial scheduling strategy corresponding to each initial scheduling strategy cluster.

[0069] In this embodiment, for each initial scheduling strategy cluster, the semantic reward function is used to calculate the semantic reward value of each first scheduling strategy in the cluster under the current production scheduling operation state. The semantic reward value represents the semantic matching degree of the strategy under the current production scheduling operation state. The first scheduling strategy with the highest semantic reward value in each cluster is selected as the representative strategy of the cluster and is used as the initial scheduling strategy. Each initial scheduling strategy cluster outputs an initial scheduling strategy, so that the multiple initial scheduling strategies obtained in subsequent steps can cover scheduling schemes with different semantic categories and avoid candidate strategies being concentrated in a single semantic direction.

[0070] Through steps 71 to 74 above, the large-scale first scheduling strategy generated by knowledge graph reasoning is semantically vectorized and clustered. Representative strategies are selected from each semantic cluster using semantic reward values ​​as initial scheduling strategies. While ensuring the semantic diversity of strategies, the size of candidate strategies is reduced, the computational complexity of subsequent reinforcement learning scoring is reduced, and the efficiency of scheduling strategy selection is improved.

[0071] In some embodiments, the first scheduling strategy, which is inferred and generated based on a preset production scheduling knowledge graph, is implemented through the following steps: Step 81: Semantically encode the current disturbance event information, calculate the cosine similarity between the encoded first semantic code and the second semantic code corresponding to the historical disturbance event information in the production scheduling knowledge graph, and select the top M historical disturbance event information based on the cosine similarity to construct a semantic evidence set.

[0072] In this embodiment, the current disturbance event information includes features such as disturbance type, occurrence time, impact range, and severity. It is mapped to a first semantic code in the vector space using a semantic coding model. The historical disturbance event information stored in the production scheduling knowledge graph is also mapped to a second semantic code using the same semantic coding model. The cosine similarity between the first semantic code and the second semantic code corresponding to each historical disturbance event is calculated. The cosine similarity is used to characterize the semantic proximity between the current disturbance event and each historical disturbance event. The higher the similarity, the closer the event type and impact pattern are. The top M historical disturbance events with the highest cosine similarity are selected to form a semantic evidence set. The semantic evidence set is used to provide historical reference for subsequent intention reasoning.

[0073] Step 82: Based on the mapping relationship between the semantic evidence set and the event intent in the production scheduling knowledge graph, calculate the semantic support strength of the current disturbance event information for the scheduling intent associated with the production scheduling knowledge graph.

[0074] In this embodiment, each historical disturbance event in the semantic evidence set has a semantic association with the scheduling intent in the production scheduling knowledge graph, and each association is assigned a weight value to represent the tendency of the event to trigger the corresponding intent. The semantic support strength of the current disturbance event for a certain scheduling intent is obtained by summing the product of the similarity between each historical disturbance event in the semantic evidence set and the current event and the corresponding event intent association weight. That is, the higher the similarity of the historical event, the greater its contribution to the support of the intent, and the higher the corresponding association weight, the greater the support strength. The semantic support strength characterizes the rationality of generating each scheduling intent under the current disturbance situation.

[0075] Step 83: Based on the state vector, perform state correction and normalization processing on the semantic support strength to generate a multi-intent set including multiple scheduling intents and the weights corresponding to each scheduling intent. Based on the multi-intent set, select the corresponding scheduling decision type from the production scheduling knowledge graph to generate a candidate strategy space. The candidate strategy space includes multiple candidate scheduling strategies, and each candidate scheduling strategy carries a corresponding intent label and state adaptation feature.

[0076] In this embodiment, the state vector includes real-time production state features such as workshop load rate, rush order ratio, and equipment availability. The same disturbance may correspond to different scheduling objectives under different workshop states. Therefore, the semantic support strength is dynamically adjusted through a state correction function so that the intent strength can reflect the impact of the current production environment. The intent strength after state correction is normalized to obtain the weight of each intent. The intent with a higher weight should be given priority under the current disturbance and workshop state. The corresponding scheduling decision type is selected from the knowledge graph according to the mapping relationship between intent and decision. Then, all candidate scheduling strategies that meet the conditions are retrieved from the knowledge graph based on the semantic constraints of the decision. Each candidate scheduling strategy carries a corresponding intent label and state adaptation feature. The intent label is used to identify the scheduling objective served by the strategy, and the state adaptation feature is used to describe the applicable working conditions of the strategy.

[0077] Step 84: After traversing all candidate scheduling strategies in the candidate strategy space and calculating the intent matching score, historical execution effect score, and state adaptability score of each candidate scheduling strategy, the intent matching score, historical execution effect score, and state adaptability score are weighted according to adaptive dynamic weights. The top N candidate scheduling strategies are selected in descending order of the weighted comprehensive score to obtain the first scheduling strategy.

[0078] In this embodiment, the intent matching score quantifies the overall support of a strategy for the current set of multiple intents by calculating the sum of the weights of the multiple intents satisfied by the strategy. When a strategy satisfies a certain intent, that intent weight is included. The historical execution effect score is based on historical scheduling cases stored in the knowledge graph, and calculates the average values ​​of indicators such as execution success rate, completion rate, and on-time delivery rate of the strategy under similar disturbance events. The state adaptability score evaluates the executability of the strategy under the current working conditions by matching the state adaptability features of the strategy with the current state vector. The adaptive dynamic weight is automatically calculated based on the current workshop state characteristics using the scenario rules stored in the knowledge graph, without manual intervention, so that the weight can be dynamically adjusted with changes in the production environment. The top N candidate scheduling strategies are selected as the first scheduling strategy after being sorted from high to low according to the comprehensive score.

[0079] Through steps 81 to 84 above, a complete reasoning chain is realized, which includes semantic retrieval, evidence aggregation, multi-intent generation, policy retrieval, and multi-dimensional scoring ranking based on the current perturbation event from the knowledge graph. Implicit scheduling experience is transformed into a computable set of candidate policies, providing a semantically reasonable and state-adaptive policy space for subsequent reinforcement learning optimization.

[0080] In some embodiments, the state vector corresponding to the production state information generated based on the current disturbance event information and the workshop environment information is determined through the following steps: Step 91: Obtain a pre-trained multi-class entity discrimination model. The multi-class entity discrimination model is trained and constructed using TF-IDF feature extraction and One-Vs-Rest linear classification strategy, with the scheduling intention and scheduling decision derived from the semantic parsing of historical scheduling text as the supervision target. The scheduling intention and scheduling decision in the supervision target are obtained by sequentially encoding the historical scheduling text with a pre-trained semantic model, performing UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering.

[0081] In this embodiment, historical scheduling text includes unstructured text such as scheduling notes, operation logs, and shift handover records. This type of text contains implicit information about the dispatcher's decision-making intentions and strategy choices. First, a pre-trained semantic model is used to encode each historical scheduling text into a high-dimensional semantic vector. Then, the UMAP dimensionality reduction method is used to map the high-dimensional semantic vector to a low-dimensional semantic space. In the low-dimensional semantic space, the HDBSCAN hierarchical density clustering method is used to automatically identify semantic clusters. The clustering process does not require a pre-set number of categories and can adaptively form semantic clusters according to the semantic density distribution, and explicitly distinguish high-density semantic regions from noise samples. For the semantic clusters formed by clustering, semantic annotation is performed through keyword extraction and representative sample screening mechanisms to summarize the scheduling intention subclass and scheduling decision subclass as supervision targets. On this basis, the TF-IDF method is used to vectorize the historical scheduling text, extract the word frequency-inverse document frequency features, and train an independent binary classifier for each semantic category using a One-Vs-Rest linear classification strategy to construct a multi-class entity discrimination model. During the inference phase, this model can output the probability value of any new input text belonging to each semantic category.

[0082] Step 92: Using a multi-category entity discrimination model, perform semantic parsing and entity extraction processing on the current disturbance event information and workshop environment information to obtain target entity instances associated with the current production state. The target entity instances include at least the following entities: equipment status entity, task status entity, material status entity, personnel status entity, order status entity, and disturbance event entity.

[0083] In this embodiment, the current disturbance event information includes unstructured descriptive text such as disturbance type, occurrence time, impact range, and severity. The workshop environment information includes multi-source heterogeneous data such as equipment operating parameters, task progress data, material inventory status, personnel scheduling information, and order delivery requirements. After the above information is uniformly organized into a text format, it is input into a multi-category entity discrimination model. The model performs semantic parsing on each text and outputs the probability distribution of the scheduling intention category and scheduling decision category to which the text belongs. By setting a probability threshold, the active entity types under the current production state are obtained. Combined with the keywords and numerical parameters extracted from the text, equipment status entities, task status entities, material status entities, personnel status entities, order status entities, and disturbance event entities are instantiated as target entity instances associated with the current production state.

[0084] Step 93: Retrieve the entity embedding vector corresponding to the target entity instance in the production scheduling knowledge graph, and concatenate the retrieved entity embedding vectors in a preset order to obtain the state vector.

[0085] In this embodiment, the production scheduling knowledge graph is pre-trained using a ConvE model for representation learning. Each entity node in the knowledge graph has been mapped to a low-dimensional continuous vector of fixed dimensions. For each target entity instance obtained in step 92, its corresponding entity embedding vector is retrieved from the production scheduling knowledge graph using the entity name as an index. The embedding vectors of six types of entities—equipment status entities, task status entities, material status entities, personnel status entities, order status entities, and disturbance event entities—are concatenated in a fixed order to obtain a state vector of uniform dimensions. The state vector integrates the semantic features of the six types of heterogeneous production status information, providing a structured and numerical description of the current production scheduling operation status for the subsequent reinforcement learning strategy network.

[0086] Through steps 91 to 93 above, semantic parsing and entity extraction of current disturbance event information and workshop environment information are achieved based on a pre-trained multi-class entity discrimination model. A unified-dimensional state vector is generated through knowledge graph retrieval and vector concatenation, which transforms unstructured production state description information into a computable structured vector representation, providing reinforcement learning models with environmental feature inputs that fit the actual production scenario.

[0087] In some embodiments, a production scheduling knowledge graph is constructed through the following steps: Step 101: Using preset semantic constraints, perform entity disambiguation and relation alignment on the acquired multi-source heterogeneous production data to obtain a semantically unified set of normalized entities and a set of normalized relations. The multi-source heterogeneous production data includes at least production status data, disturbance event data, historical scheduling strategy data, and scheduling experience text.

[0088] In this embodiment, the multi-source heterogeneous production data originates from heterogeneous platforms such as Manufacturing Execution System (MES), Enterprise Resource Planning (ERP), and Equipment Management System (EMS), and there is a lack of unified semantic association between different data sources. The preset semantic constraints are based on a pre-built production scheduling ontology model, which standardizes the core concepts and their relationships, such as workshop status, disturbance events, scheduling intentions, scheduling decisions, and scheduling strategies. Entity disambiguation maps different expressions pointing to the same real object from different data sources to the same entity instance. For example, different expressions such as "equipment A," "machine A," and "machine tool A" all point to the same equipment entity. Relation alignment maps semantically equivalent but diverse predicates to a unified set of normalized relations. For example, different predicates such as "cause," "lead to," and "trigger" are unified into causal relations. After the above processing, a semantically unified set of normalized entities and a set of normalized relations are obtained.

[0089] Step 102: Using a pre-defined implicit knowledge extraction model, implicit knowledge is extracted from the scheduling experience text to obtain a structured scheduling intention ontology, scheduling decision ontology, and corresponding scheduling intention subclass instances and scheduling decision subclass instances. The implicit knowledge extraction model is trained and constructed using TF-IDF feature extraction and One-Vs-Rest linear classification strategies, with the conceptual hierarchy of the scheduling intention ontology and scheduling decision ontology derived from the semantic parsing of historical scheduling experience text as the supervised objective. The conceptual hierarchy of the scheduling intention ontology and scheduling decision ontology in the supervised objective is obtained by unsupervised clustering of historical scheduling experience text, combining pre-trained semantic model encoding, UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering.

[0090] In this embodiment, the scheduling experience text includes unstructured text such as scheduling notes, operation logs, and shift handover records. This type of text implicitly contains the dispatcher's decision-making intentions and strategy selection information but lacks explicit structured markings. The implicit knowledge extraction process first performs pre-trained semantic model encoding, UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering on the historical scheduling experience text. Semantic clusters are automatically identified without pre-setting the number of categories. A recursive refinement mechanism is used to achieve hierarchical division from coarse-grained semantic categories to fine-grained subcategories, summarizing the conceptual hierarchy of the scheduling intention ontology and the scheduling decision ontology. Based on this, the historical scheduling experience text is vectorized using the summarized scheduling intention ontology and scheduling decision ontology as supervision targets. The TF-IDF method is used to train independent binary classifiers for each semantic category using a One-Vs-Rest linear classification strategy, constructing an implicit knowledge extraction model. During the inference phase, this model can output the scheduling intention subclass instance and scheduling decision subclass instance to which the input scheduling experience text belongs, realizing the structured expression of implicit knowledge.

[0091] Step 103 involves associating and mapping the normalized entity set, normalized relation set, scheduling intent ontology, scheduling decision ontology, scheduling intent subclass instance, and scheduling decision subclass instance into temporal quadruple knowledge facts including effective time intervals, and mapping the temporal quadruple knowledge facts to the graph database to generate the initial graph.

[0092] In this embodiment, the normalized entity set and the normalized relation set constitute the basic instance layer of the knowledge graph, the scheduling intention ontology and the scheduling decision ontology constitute the conceptual layer constraints of the knowledge graph, and the scheduling intention subclass instance and the scheduling decision subclass instance connect the conceptual layer and the instance layer; the temporal quadruple is represented in the form of (h, r, t, τ), where h represents the head entity, r represents the relation, t represents the tail entity, and τ represents the effective time interval of the knowledge fact; by converting various entities, relations and ontology structures into temporal quadruple knowledge facts, the knowledge facts are mapped to the graph database, entities are mapped to nodes in the graph database, relations are mapped to edges between nodes, entity attributes are mapped to attribute fields of nodes, and ontology hierarchical relations are mapped to type constraints of edges, thereby generating the initial graph.

[0093] Step 104 is repeated to update old knowledge facts with new knowledge facts to iteratively update the initial graph until the production scheduling knowledge graph is obtained.

[0094] In this embodiment, the process of repeatedly updating old knowledge facts with newly created ones is repeated to iteratively update the initial knowledge graph until a production scheduling knowledge graph is obtained. When the workshop state or a disturbance event changes, the system receives the change input, first locates the affected old knowledge facts and sets the end time of its effective time interval to the current time, rendering the original knowledge facts invalid. Then, a new knowledge fact is created with the changed new value and its start time is set to the current time. Incremental updates to the knowledge graph are achieved through a time-sequential revision method of old knowledge fact invalidation and new knowledge fact insertion. The iterative update process continues, enabling the knowledge graph to continuously absorb real-time production data and maintain consistency with the actual production scenario, ultimately resulting in a production scheduling knowledge graph that integrates implicit scheduling experience and dynamic workshop information.

[0095] It should be noted that the technical means, algorithms, and data processing methods used in the process of generating the production scheduling knowledge graph, such as knowledge extraction (entity extraction, relation extraction, attribute extraction), knowledge fusion, knowledge processing (evaluation, reasoning), and knowledge updating, are clear and implementable to those skilled in the art, and the production scheduling knowledge graph can be generated through the corresponding processing methods in the existing technology.

[0096] Through steps 101 to 104 above, a complete construction process from multi-source heterogeneous production data to a time-series knowledge graph is realized. Semantic unification of multi-source data is achieved through entity disambiguation and relation alignment. Structured expression of unstructured experience text is realized through implicit knowledge extraction model. Persistent storage of knowledge is realized through mapping time-series quadruples to graph database. Synchronous evolution of the graph and dynamic production environment is realized through time-series update mechanism.

[0097] This embodiment also provides a production scheduling strategy screening device based on reinforcement learning, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0098] Figure 3 This is a structural block diagram of a production scheduling strategy screening device based on reinforcement learning according to an embodiment of this application, as shown below. Figure 3 As shown, the device includes a determining module 31, a selecting module 32, a scoring module 33, and a processing module 34, wherein, The determination module 31 is used to determine the state vector corresponding to the production state information generated based on the current disturbance event information and workshop environment information after acquiring multiple initial scheduling strategies. The production state information is used to characterize the production scheduling operation status under the influence of the disturbance event. The selection module 32 is coupled to the determination module 31 and is used to perform reinforcement learning scoring on multiple initial scheduling strategies based on a preset multilayer perceptron (MLP) and state vectors, obtain the probability distribution of the initial scheduling strategies based on the scores, and select multiple candidate scheduling strategies from the multiple initial scheduling strategies based on the probability distribution. The scoring module 33, coupled to the selection module 32, is used to determine the performance semantic quantitative score of the candidate scheduling strategy according to the preset performance semantic reward function. The performance semantic reward function is constructed based on the scheduling performance reward function and the semantic reward function. The scheduling performance reward function is used to calculate the expected performance parameters corresponding to the candidate scheduling strategy, and the semantic reward function is used to calculate the semantic matching degree between the state vector and the candidate scheduling strategy. The processing module 34, coupled to the scoring module 33, is used to select the candidate scheduling strategies ranked in the top K by performance semantic quantification score from the candidate scheduling strategies, and obtain the screening result including the top K candidate scheduling strategies, where K is a positive integer not greater than the number of candidate scheduling strategies.

[0099] In some embodiments, the selection module 32 further includes: The concatenation unit is used to determine the scheduling policy semantic vector corresponding to each initial scheduling policy, and concatenate all scheduling policy semantic vectors with the state vector to obtain the fused feature vector corresponding to each initial scheduling policy.

[0100] The scoring unit, coupled to the concatenation unit, is used to learn the interaction features of the fused feature vector using a multilayer perceptron (MLP), and to score the interaction features according to the scoring function corresponding to the MLP to obtain a matching score. The scoring function is constructed based on the network parameters obtained by reinforcement learning based on a preset activation function. The network parameters are used to represent the weight matrix and bias vector of all layers in the MLP.

[0101] In some embodiments, the selection module 32 further includes: The transformation unit is used to process the matching score using a normalized exponential function to obtain the probability distribution corresponding to all initial scheduling strategies.

[0102] The first determining unit, coupled to the transformation unit, is used to determine the matching probability corresponding to each initial scheduling strategy in the probability distribution, wherein the matching probability is used to characterize the probability of selecting an initial scheduling strategy in the production scheduling operation state.

[0103] The filtering unit, coupled to the first determining unit, is used to filter initial scheduling strategies from multiple initial scheduling strategies based on the matching probability, such that the matching probability is greater than a preset probability threshold, to obtain multiple candidate scheduling strategies.

[0104] In some embodiments, the scoring module 33 further includes: The second determining unit is used to determine the expected performance parameter value corresponding to the candidate scheduling strategy using the scheduling performance reward function, and to determine the semantic reward parameter value corresponding to the candidate scheduling strategy according to the semantic reward function.

[0105] The weighting unit, coupled to the second determining unit, is used to weight the expected performance parameter value and the semantic reward parameter value according to the performance semantic reward function to obtain the performance semantic quantitative score corresponding to the candidate scheduling strategy.

[0106] In some embodiments, the second determining unit is further configured to acquire a state performance mapping parameter table associated with the scheduling performance reward function. The state performance mapping parameters include scheduling strategies, expected performance, and the correspondence between them. The state performance mapping parameter table is generated based on historical scheduling strategies and corresponding historical scheduling response performance data under preset states. The scheduling strategy is associated with corresponding production state information, and the historical scheduling response performance data is associated with multiple target evaluation indicators: total delay time, average completion time, and delivery rate. The expected performance corresponding to the candidate scheduling strategy is detected in the state performance mapping parameter table. The expected performance is then subjected to deviation standardization processing using the scheduling performance reward function to obtain the expected performance parameter values ​​corresponding to the candidate scheduling strategy.

[0107] In some embodiments, the second determining unit is further configured to perform word vector processing on the policy text of the candidate scheduling policy to obtain a first policy feature vector corresponding to the candidate scheduling policy; calculate the semantic matching degree corresponding to the first policy feature vector and the state vector, and process the semantic matching degree using a preset semantic reward function to generate a semantic reward value corresponding to the semantic matching degree, wherein the semantic reward parameter value includes the semantic reward value, and the semantic reward function uses the Sigmoid function to perform semantic matching degree mapping transformation.

[0108] In some embodiments, before acquiring multiple initial scheduling strategies, the reinforcement learning-based production scheduling strategy screening device is further configured to acquire a first scheduling strategy generated by reasoning based on a preset production scheduling knowledge graph, wherein the first scheduling strategy includes strategy text; map the strategy text to a second strategy feature vector using a ConvE model, and encode the strategy semantic vector corresponding to the first scheduling strategy according to the second strategy feature vector; calculate the cosine similarity between any two first scheduling strategies based on the strategy semantic vector, and cluster all first scheduling strategies according to the cosine similarity to obtain multiple initial scheduling strategy clusters, wherein the initial scheduling strategy clusters include at least two first scheduling strategies with a cosine similarity greater than a set similarity threshold; determine the semantic reward value of each first scheduling strategy in the initial scheduling strategy cluster according to the semantic reward function, and select the first scheduling strategy with the highest semantic reward value from each initial scheduling strategy cluster to obtain the initial scheduling strategy corresponding to each initial scheduling strategy cluster.

[0109] In some embodiments, the reinforcement learning-based production scheduling strategy screening device is further configured to semantically encode the current disturbance event information, calculate the cosine similarity between the encoded first semantic code and the second semantic code corresponding to the historical disturbance event information in the production scheduling knowledge graph, and select the top M historical disturbance event information based on the cosine similarity to construct a semantic evidence set; calculate the semantic support strength of the current disturbance event information for the scheduling intent associated with the production scheduling knowledge graph based on the mapping relationship between the semantic evidence set and the event intent in the production scheduling knowledge graph; and perform state correction and normalization processing on the semantic support strength based on the state vector to generate a set including multiple scheduling intents and the corresponding... A weighted set of multiple intentions is generated, and based on this set, corresponding scheduling decision types are selected from the production scheduling knowledge graph to generate a candidate strategy space. This space includes multiple candidate scheduling strategies, each carrying a corresponding intention tag and state adaptation feature. After traversing all candidate scheduling strategies in the space and calculating the intention matching score, historical execution effect score, and state adaptation score for each strategy, the intention matching score, historical execution effect score, and state adaptation score are weighted according to adaptive dynamic weights. The top N candidate scheduling strategies are then selected in descending order of the weighted comprehensive score to obtain the first scheduling strategy.

[0110] In some embodiments, the determining module 31 further includes: The acquisition unit is used to acquire a pre-trained multi-class entity discrimination model. The multi-class entity discrimination model is trained and constructed using TF-IDF feature extraction and One-Vs-Rest linear classification strategies, with the scheduling intention and scheduling decision derived from the semantic parsing of historical scheduling text as the supervision target. The scheduling intention and scheduling decision in the supervision target are obtained by sequentially encoding the historical scheduling text with a pre-trained semantic model, performing UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering.

[0111] The extraction unit, coupled to the acquisition unit, is used to perform semantic parsing and entity extraction processing on the current disturbance event information and workshop environment information using a multi-category entity discrimination model to obtain target entity instances associated with the current production state. The target entity instances include at least the following entities: equipment status entity, task status entity, material status entity, personnel status entity, order status entity, and disturbance event entity.

[0112] The generation unit, coupled with the extraction unit, is used to retrieve the entity embedding vector corresponding to the target entity instance in the production scheduling knowledge graph, and to concatenate the retrieved entity embedding vectors in a preset order to obtain the state vector.

[0113] In some embodiments, the reinforcement learning-based production scheduling strategy screening device further constructs a production scheduling knowledge graph through the following steps: Using preset semantic constraints, entity disambiguation and relation alignment are performed on the acquired multi-source heterogeneous production data to obtain a semantically unified standardized entity set and a standardized relation set. The multi-source heterogeneous production data includes at least production status data, disturbance event data, historical scheduling strategy data, and scheduling experience text. Using a preset implicit knowledge extraction model, implicit knowledge is extracted from the scheduling experience text to obtain a structured scheduling intent ontology, scheduling decision ontology, and corresponding scheduling intent subclass instances and scheduling decision subclass instances. The implicit knowledge extraction model uses the conceptual hierarchy of the scheduling intent ontology and scheduling decision ontology derived from the semantic parsing and summarization of historical scheduling experience text as the supervision target, and employs TF-IDF feature extraction and One-Vs-R. The concept hierarchy of the scheduling intention ontology and scheduling decision ontology in the supervised objective is constructed by training a linear classification strategy. It is obtained by unsupervised clustering of historical scheduling experience texts, combining pre-trained semantic model encoding, UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering. The normalized entity set, normalized relation set, scheduling intention ontology, scheduling decision ontology, scheduling intention subclass instances, and scheduling decision subclass instances are associated and structurally mapped to convert them into temporal quadruples of knowledge facts including effective time intervals. The temporal quadruples of knowledge facts are then mapped to a graph database to generate an initial graph. The process is repeated to update the old knowledge facts with the new knowledge facts to iteratively update the initial graph until the production scheduling knowledge graph is obtained.

[0114] This embodiment also provides a production scheduling strategy screening system, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0115] Optionally, the production scheduling strategy screening system may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0116] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0117] S1, after obtaining multiple initial scheduling strategies, determines the state vector corresponding to the production state information generated based on the current disturbance event information and workshop environment information, wherein the production state information is used to characterize the production scheduling operation state under the influence of disturbance events.

[0118] S2, based on the preset multilayer perceptron (MLP) and state vector, performs reinforcement learning scoring on multiple initial scheduling strategies, obtains the probability distribution of the initial scheduling strategies based on the scores, and selects multiple candidate scheduling strategies from the multiple initial scheduling strategies based on the probability distribution.

[0119] S3. Based on the preset performance semantic reward function, determine the performance semantic quantitative score of the candidate scheduling strategy. The performance semantic reward function is constructed based on the scheduling performance reward function and the semantic reward function. The scheduling performance reward function is used to calculate the expected performance parameters corresponding to the candidate scheduling strategy, and the semantic reward function is used to calculate the semantic matching degree between the state vector and the candidate scheduling strategy.

[0120] S4. From the candidate scheduling strategies, select the candidate scheduling strategies ranked in the top K by performance semantic quantification score to obtain the screening results including the top K candidate scheduling strategies, where K is a positive integer not greater than the number of candidate scheduling strategies.

[0121] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0122] Furthermore, in conjunction with the reinforcement learning-based production scheduling strategy selection method in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the reinforcement learning-based production scheduling strategy selection methods in the above embodiments.

[0123] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A production scheduling strategy selection method based on reinforcement learning, characterized in that, include: After acquiring multiple initial scheduling strategies, a state vector corresponding to the production status information generated based on the current disturbance event information and workshop environment information is determined, wherein the production status information is used to characterize the production scheduling operation status under the influence of the disturbance event; Based on a preset multilayer perceptron (MLP) and the state vector, reinforcement learning scores are applied to multiple initial scheduling strategies, and the probability distribution of the initial scheduling strategies is obtained based on the scores. Based on the probability distribution, multiple candidate scheduling strategies are selected from the multiple initial scheduling strategies. According to the preset performance semantic reward function, the performance semantic quantitative score of the candidate scheduling strategy is determined. The performance semantic reward function is constructed based on the scheduling performance reward function and the semantic reward function. The scheduling performance reward function is used to calculate the expected performance parameters corresponding to the candidate scheduling strategy, and the semantic reward function is used to calculate the semantic matching degree between the state vector and the candidate scheduling strategy. From the candidate scheduling strategies, the candidate scheduling strategies ranked in the top K by the performance semantic quantization score are selected to obtain the screening result including the top K candidate scheduling strategies, where K is a positive integer not greater than the number of candidate scheduling strategies.

2. The method according to claim 1, characterized in that, Based on a preset multilayer perceptron (MLP) and the state vector, reinforcement learning scoring is performed on multiple initial scheduling policies, including: Determine the scheduling strategy semantic vector corresponding to each initial scheduling strategy, and concatenate all the scheduling strategy semantic vectors with the state vector to obtain the fusion feature vector corresponding to each initial scheduling strategy; Using the Multilayer Perceptron (MLP), the interaction features of the fused feature vector are learned, and the interaction features are scored according to the scoring function corresponding to the MLP to obtain a matching score. The scoring function is constructed based on the network parameters obtained by reinforcement learning based on a preset activation function. The network parameters are used to represent the weight matrix and bias vector of all layers in the MLP.

3. The method according to claim 2, characterized in that, The probability distribution of the initial scheduling strategy is obtained based on the scoring, and multiple candidate scheduling strategies are selected from the multiple initial scheduling strategies based on the probability distribution, including: The matching score is processed using a normalized exponential function to obtain the probability distribution corresponding to all the initial scheduling strategies; In the probability distribution, a matching probability is determined for each initial scheduling strategy, wherein the matching probability is used to characterize the probability of selecting the initial scheduling strategy in the production scheduling operation state; Based on the matching probability, the initial scheduling strategies with matching probabilities greater than a preset probability threshold are selected from the multiple initial scheduling strategies to obtain multiple candidate scheduling strategies.

4. The method according to claim 1, characterized in that, Based on a preset performance semantic reward function, the performance semantic quantitative score of the candidate scheduling strategy is determined, including: The expected performance parameter value corresponding to the candidate scheduling strategy is determined using the scheduling performance reward function, and the semantic reward parameter value corresponding to the candidate scheduling strategy is determined using the semantic reward function. The expected performance parameter value and the semantic reward parameter value are weighted according to the performance semantic reward function to obtain the performance semantic quantitative score corresponding to the candidate scheduling strategy.

5. The method according to claim 4, characterized in that, Using the scheduling performance reward function, the expected performance parameters corresponding to the candidate scheduling strategy are determined, including: Obtain a state performance mapping parameter table associated with the scheduling performance reward function, wherein the state performance mapping parameters include scheduling strategy, expected performance and the correspondence between the two, the state performance mapping parameter table is generated based on historical scheduling strategies and corresponding historical scheduling response performance data under preset states, the scheduling strategy is associated with corresponding production state information, and the historical scheduling response performance data is associated with the following multiple target evaluation indicators: total delay time, average completion time, and delivery rate; In the state performance mapping parameter table, detect the expected performance corresponding to the candidate scheduling strategy; The expected performance is standardized by performing deviation processing on the scheduling performance reward function to obtain the expected performance parameter value corresponding to the candidate scheduling strategy.

6. The method according to claim 4, characterized in that, Based on the semantic reward function, the semantic reward parameter values ​​corresponding to the candidate scheduling strategy are determined, including: The policy text of the candidate scheduling policy is processed by word vectors to obtain the first policy feature vector corresponding to the candidate scheduling policy; Calculate the semantic matching degree between the first policy feature vector and the state vector, and process the semantic matching degree using the preset semantic reward function to generate a semantic reward value corresponding to the semantic matching degree. The semantic reward parameter value includes the semantic reward value, and the semantic reward function uses the Sigmoid function to perform semantic matching degree mapping transformation.

7. The method according to claim 1, characterized in that, Before obtaining multiple initial scheduling policies, the method includes: Obtain a first scheduling strategy generated by reasoning based on a preset production scheduling knowledge graph, wherein the first scheduling strategy includes strategy text; The policy text is mapped to a second policy feature vector using the ConvE model, and the policy semantic vector corresponding to the first scheduling policy is encoded based on the second policy feature vector. Based on the policy semantic vector, the cosine similarity between any two first scheduling policies is calculated, and all first scheduling policies are clustered according to the cosine similarity to obtain multiple initial scheduling policy clusters. The initial scheduling policy clusters include at least two first scheduling policies whose cosine similarity is greater than a set similarity threshold. Based on the semantic reward function, the semantic reward value of each first scheduling strategy in the initial scheduling strategy cluster is determined, and the first scheduling strategy with the highest semantic reward value is selected from each initial scheduling strategy cluster to obtain the initial scheduling strategy corresponding to each initial scheduling strategy cluster.

8. The method according to claim 7, characterized in that, Based on a pre-defined production scheduling knowledge graph, the first scheduling strategy generated through reasoning includes: The current disturbance event information is semantically encoded, the cosine similarity between the encoded first semantic code and the second semantic code corresponding to the historical disturbance event information in the production scheduling knowledge graph is calculated, and the top M historical disturbance event information with the highest cosine similarity is selected to construct a semantic evidence set. Based on the semantic evidence set and the event intent mapping relationship in the production scheduling knowledge graph, calculate the semantic support strength of the current disturbance event information for the scheduling intent associated with the production scheduling knowledge graph; Based on the state vector, the semantic support strength is subjected to state correction and normalization processing to generate a multi-intent set including multiple scheduling intents and the weight corresponding to each scheduling intent. Based on the multi-intent set, the corresponding scheduling decision type is selected from the production scheduling knowledge graph to generate a candidate strategy space. The candidate strategy space includes multiple candidate scheduling strategies, and the candidate scheduling strategies carry corresponding intent tags and state adaptation features. After traversing all the candidate scheduling strategies in the candidate strategy space and calculating the intent matching score, historical execution effect score, and state adaptability score of each candidate scheduling strategy, the intent matching score, historical execution effect score, and state adaptability score are weighted according to adaptive dynamic weights, and the top N candidate scheduling strategies are selected in descending order of the weighted comprehensive score to obtain the first scheduling strategy.

9. The method according to claim 8, characterized in that, Determine the state vector corresponding to the production state information generated based on the current disturbance event information and workshop environment information, including: A pre-trained multi-class entity discrimination model is obtained, wherein the multi-class entity discrimination model is trained and constructed using TF-IDF feature extraction and One-Vs-Rest linear classification strategy, with the scheduling intention and scheduling decision derived from the semantic parsing of historical scheduling text as the supervision target; the scheduling intention and scheduling decision in the supervision target are obtained by sequentially performing pre-trained semantic model encoding, UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering on the historical scheduling text. Using the multi-category entity discrimination model, semantic parsing and entity extraction are performed on the current disturbance event information and the workshop environment information to obtain target entity instances associated with the current production state. The target entity instances include at least the following entities: equipment status entity, task status entity, material status entity, personnel status entity, order status entity, and disturbance event entity. The entity embedding vector corresponding to the target entity instance is retrieved from the production scheduling knowledge graph, and the retrieved entity embedding vectors are concatenated in a preset order to obtain the state vector.

10. The method according to claim 7, characterized in that, Constructing the production scheduling knowledge graph includes: Using preset semantic constraints, entity disambiguation and relation alignment are performed on the acquired multi-source heterogeneous production data to obtain a semantically unified set of normalized entities and a set of normalized relations. The multi-source heterogeneous production data includes at least production status data, disturbance event data, historical scheduling strategy data, and scheduling experience text. Using a pre-defined implicit knowledge extraction model, implicit knowledge is extracted from the scheduling experience text to obtain a structured scheduling intention ontology, scheduling decision ontology, and corresponding scheduling intention subclass instances and scheduling decision subclass instances. The implicit knowledge extraction model is trained and constructed using TF-IDF feature extraction and One-Vs-Rest linear classification strategies, with the hierarchical structure of the scheduling intention ontology and scheduling decision ontology derived from semantic parsing of historical scheduling experience text as the supervised objective. The hierarchical structure of the scheduling intention ontology and scheduling decision ontology in the supervised objective is obtained by unsupervised clustering of the historical scheduling experience text, combining pre-trained semantic model encoding, UMAP dimensionality reduction, and HDBSCAN hierarchical density clustering. The normalized entity set, the normalized relation set, the scheduling intent ontology, the scheduling decision ontology, the scheduling intent subclass instance, and the scheduling decision subclass instance are associated and structurally mapped to be converted into time-series quadruple knowledge facts including effective time intervals. The time-series quadruple knowledge facts are then mapped to a graph database to generate an initial graph. Repeatedly update the old knowledge facts with the new knowledge facts to iteratively update the initial graph until the production scheduling knowledge graph is obtained.

Images