Graph neural network-based reinforcement learning detection production line scheduling method and apparatus

Abstract

The invention relates to a reinforcement learning detection production line scheduling method and device based on a graph neural network. The method comprises the following steps: acquiring a node state quantity of an equipment node; combining the node state quantities of the equipment nodes and the materials into an equipment node set, and mapping the state of the equipment node set to a uniform-dimension feature space by using an independent full-connection layer, so as to obtain a node embedding vector; splicing the embedded feature vectors of all the nodes in sequence to form a node feature matrix; after the node feature matrix is transformed, a self-adaptive directed adjacency matrix is obtained; obtaining a final node through the node feature matrix and the adaptive directed adjacency matrix; and carrying out production scheduling on the detection production line according to the scheduling strategy, modeling the earphone acoustic detection production line into a graph structure with a dynamic directed interaction relationship, extracting global correlation characteristics among equipment nodes by using a graph neural network, and carrying out production scheduling on the detection production line according to the global correlation characteristics. And a deep reinforcement learning algorithm is combined to realize autonomous learning and optimization decision of the intelligent scheduling strategy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a reinforcement learning detection production line scheduling method based on graph neural networks, a reinforcement learning detection production line scheduling device based on graph neural networks, a computer device, and a storage medium. Background Technology

[0002] Current headphone acoustic testing production lines generally employ manually designed fixed scheduling strategies, controlling the testing process through preset operation sequences and execution logic. While this method can meet basic production needs in some scenarios, it has significant shortcomings in complex industrial environments. First, the acoustic testing process for headphones involves multiple parallel devices and procedures, and the combination space of scheduling strategies increases exponentially with the number of devices and batches of materials. Manually designed strategies are difficult to exhaustively optimize the high-dimensional strategy space, and cannot ensure that the scheduling scheme reaches the global optimum, thus limiting further improvements in testing efficiency.

[0003] Furthermore, both the start-up position and the inspection box exhibit processing time fluctuations, which are affected by equipment status and system load. Fixed scheduling strategies cannot perceive the dynamic status of the production line in real time and lack the ability to adjust decisions based on environmental changes, easily leading to suboptimal operating conditions such as equipment idleness and task backlog, resulting in a decrease in overall efficiency.

[0004] On the other hand, during long-term operation, equipment failures such as detection box shutdowns or gripper jamming are quite common. Manually preset strategies are difficult to maintain maximum capacity in the event of partial equipment failure, and may even lead to situations where the scheduling process cannot continue, requiring manual revision of rules or adjustment of on-site configurations, further affecting production continuity.

[0005] To overcome the shortcomings of fixed manual strategies, some research in recent years has begun to explore the use of deep reinforcement learning techniques for automated scheduling. However, existing reinforcement learning models generally use a uniform encoding method to process environmental states, simply concatenating the states of all devices and directly inputting them into the neural network for action reasoning. This lacks a structured representation of the operational dependencies between devices in the headphone acoustic testing production line, making it difficult to effectively identify key scheduling factors that affect overall efficiency. Consequently, policy decisions still suffer from performance degradation and instability under complex operating conditions. Summary of the Invention

[0006] In view of the above problems, embodiments of the present invention are proposed to provide a graph neural network-based reinforcement learning detection production line scheduling method, a graph neural network-based reinforcement learning detection production line scheduling device, a computer device, and a storage medium to overcome or at least partially solve the above problems.

[0007] To achieve the above objectives, this invention proposes a reinforcement learning-based production line scheduling detection method based on graph neural networks, the method comprising: Obtain the node status of equipment nodes and materials; The node state variables of the equipment nodes and materials are combined into a set of equipment nodes. An independent fully connected layer is used to map the state of the set of equipment nodes to a feature space of a unified dimension, thereby obtaining the node embedding vector. The embedded feature vectors of all nodes are concatenated in order to form a node feature matrix; After transforming the node feature matrix, an adaptive directed adjacency matrix is ​​obtained; The final node is obtained through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; Select the sub-action corresponding to the maximum Q value as the optimal decision in the current state; The current device state is transitioned according to the selected sub-action to obtain the node state quantity at the next moment. This process is repeated until the scheduling strategy of the detection production line is output. The production of the detection production line is then scheduled according to the scheduling strategy.

[0008] Preferably, the method includes an agent performing an action based on the current state, and the environment returning a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

[0009] Preferably, the method further includes: Obtain instant rewards, discount factors, current status, current action, next status, online network parameters, and target network parameters; The time difference error is obtained based on the instant reward, discount factor, current state, current action, next state, online network parameters, and target network parameters.

[0010] Preferably, the node status quantities of the equipment nodes and materials include the node status quantities corresponding to the robotic arm, gripper, start-up position, and detection box.

[0011] Preferably, the step of transforming the node feature matrix to obtain the adaptive directed adjacency matrix includes: The node feature matrix is ​​mapped by a learnable linear transformation matrix to obtain the transformed node; Calculate the attention relevance score between any two nodes, and perform softmax processing on all associated edges of each node to form an adaptive directed adjacency matrix.

[0012] Preferably, the method further includes: The device characteristics are defined as a set of nodes in a directed graph; The adaptive directed adjacency matrix is ​​defined as the set of edges of the directed graph.

[0013] This invention proposes a reinforcement learning-based production line scheduling detection device based on graph neural networks, the device comprising: The acquisition module is used to acquire the node status quantities of equipment nodes and materials; The node embedding vector module is used to combine the node state variables of the device nodes and materials into a device node set, and use an independent fully connected layer to map the state of the device node set to a feature space of a unified dimension, thereby obtaining the node embedding vector. The node feature matrix module is used to concatenate the embedded feature vectors of all nodes in sequence to form a node feature matrix; The directed adjacency matrix module is used to obtain an adaptive directed adjacency matrix after transforming the node feature matrix. The final node module is used to obtain the final node through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; The optimal decision module is used to select the sub-action corresponding to the maximum Q value as the optimal decision in the current state. The production scheduling module is used to perform state transitions on the current equipment state based on the selected sub-actions, obtain the node state quantity at the next moment, and iterate in a loop until the scheduling strategy of the inspection production line is output, and to perform production scheduling on the inspection production line according to the scheduling strategy.

[0014] Preferably, the device includes an intelligent agent that performs actions based on the current state, and the environment returns a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

[0015] This invention discloses a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described reinforcement learning detection production line scheduling method based on graph neural networks.

[0016] This invention discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based production line scheduling method based on graph neural networks.

[0017] In this embodiment of the invention, the reinforcement learning-based detection production line scheduling method based on graph neural networks includes: acquiring the node state variables of equipment nodes and materials; combining the node state variables of the equipment nodes and materials into a set of equipment nodes, and using an independent fully connected layer to map the state of the set of equipment nodes to a feature space of a unified dimension, thereby obtaining node embedding vectors; concatenating the embedding feature vectors of all nodes in sequence to form a node feature matrix; transforming the node feature matrix to obtain an adaptive directed adjacency matrix; obtaining the final node through the node feature matrix, the adaptive directed adjacency matrix, and a learnable linear transformation matrix; selecting the sub-action corresponding to the maximum Q value as the optimal decision in the current state; performing state transition on the current equipment state according to the selected sub-action to obtain the node state variable at the next moment, iterating cyclically until the scheduling strategy of the detection production line is output; and performing production scheduling on the detection production line according to the scheduling strategy. By modeling the headphone acoustic detection production line as a graph structure with dynamic directed interaction relationships, the graph neural network is used to extract the global correlation features between each equipment node, and a deep reinforcement learning algorithm is combined to realize the autonomous learning and optimization decision of the intelligent scheduling strategy. By independently embedding different device nodes, the model can fully express the heterogeneous attributes between devices; by modeling the directed interaction relationships between devices through the attention mechanism, the agent can dynamically capture the dependencies between production line devices; through multi-layer graph feature aggregation and composite reward function optimization, the system can achieve stable and efficient scheduling decisions under non-ideal working conditions such as processing time jitter, equipment abnormalities and complex task switching, thereby significantly improving the overall detection efficiency and robustness of the acoustic inspection production line. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of an embodiment of a reinforcement learning-based production line scheduling method for graph neural networks according to the present invention. Figure 2 This is a schematic diagram of a sorting and testing production line equipment configuration according to an embodiment of the present invention; Figure 3 This is a system architecture diagram of a reinforcement learning detection production line scheduling system based on graph neural networks according to an embodiment of the present invention; Figure 4 This is a structural block diagram of an embodiment of a reinforcement learning detection production line scheduling device based on graph neural networks according to an embodiment of the present invention; Figure 5This is an internal structural diagram of a computer device according to one embodiment. Detailed Implementation

[0020] To make the technical problems, technical solutions, and beneficial effects solved by the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0021] Reference Figure 1 This document illustrates a flowchart of an embodiment of a reinforcement learning-based production line scheduling method based on graph neural networks according to the present invention. The method may specifically include the following steps: Step 101: Obtain the node status of the equipment node and the material; In this embodiment of the invention, the reinforcement learning detection production line scheduling method can be applied to the automation application scenario of headphone acoustic detection production line. Of course, it can also be applied to other hardware detection production lines, such as smartphones, smart tablets, etc. This embodiment of the invention does not impose too many restrictions on the types of hardware in the detection production line. In one specific example, the production line diagram used in this invention is as follows: Figure 2 As shown, the main equipment configuration of this production line includes a robotic arm (equipped with two pairs of grippers, namely gripper jaw1, gripper jaw2, gripper jaw3 and gripper jaw4), a test tray conveyor belt M0, a test tray M1, a power-on position M2, a test box M3, a test box M4, a test box M5, a test box M6, a good product tested tray conveyor belt M7, and a defective product tested material conveyor belt M8. These are used to complete operations such as headphone power-on, acoustic testing, and sorting of test results, realizing the automation, efficiency, and intelligent operation of the entire headphone testing process.

[0022] Step 102: Combine the node state variables of the device nodes and materials into a device node set, and use an independent fully connected layer to map the state of the device node set to a feature space of a unified dimension, thereby obtaining the node embedding vector. Step 103: Concatenate the embedded feature vectors of all nodes in order to form a node feature matrix; Step 104: After transforming the node feature matrix, an adaptive directed adjacency matrix is ​​obtained; Step 105: Obtain the final node through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; Step 106: Select the sub-action corresponding to the maximum Q value as the optimal decision in the current state; Step 107: Based on the selected sub-action, perform state transition on the current device state to obtain the node state quantity at the next moment. Iterate in a loop until the scheduling strategy of the detection production line is output, and perform production scheduling on the detection production line according to the scheduling strategy.

[0023] Preferably, the method includes an agent performing an action based on the current state, and the environment returning a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

[0024] Further applied to embodiments of the present invention, the method further includes: Obtain instant rewards, discount factors, current status, current action, next status, online network parameters, and target network parameters; The time difference error is obtained based on the instant reward, discount factor, current state, current action, next state, online network parameters, and target network parameters.

[0025] In this embodiment of the invention, the node status quantities of the equipment nodes and materials include the node status quantities corresponding to the robotic arm, gripper, start-up position, and detection box.

[0026] In a further embodiment of the present invention, the step of obtaining an adaptive directed adjacency matrix after transformation through the node feature matrix includes: The node feature matrix is ​​mapped by a learnable linear transformation matrix to obtain the transformed node; Calculate the attention relevance score between any two nodes, and perform softmax processing on all associated edges of each node to form an adaptive directed adjacency matrix.

[0027] Specifically, in this embodiment of the invention, the method further includes: determining the device features as a set of nodes in a directed graph; and determining the adaptive directed adjacency matrix as a set of edges in a directed graph.

[0028] Reference Figure 3 This diagram illustrates the architecture of a reinforcement learning-based detection production line scheduling method based on graph neural networks, according to an embodiment of the present invention. The scheme first constructs a virtual simulation environment consistent with an actual headphone acoustic testing production line for the training and verification of the intelligent agent. Within this virtual environment, the real-time operating state of each key piece of equipment in the production line is represented as a state vector. This serves as the state space input for the agent. Based on the state space information, the agent uses a deep learning model to calculate the Q-values ​​of each candidate sub-action in the action space, and then selects the sub-action with the maximum Q-value. As the current state The optimal scheduling decision is then made by the virtual environment based on the selected action. Perform a state transition on the current device state to obtain the state at the next moment. By continuously executing the interactive loop of "state-decision-state update" in a virtual environment, the agent can gradually learn efficient scheduling strategies applicable to actual production lines. The detailed operation process of this solution is as follows: First, the key equipment in the headphone acoustic testing production line is abstracted as a set of discrete nodes, and their states constitute the input space of the agent. Let the set of equipment nodes be represented as... To achieve real-time awareness of device operating modes and resource status, this solution defines each device node. The state feature vector is: (1) in, The device's operating status (such as idle, busy, fault) is represented discretely using One-Hot encoding. This indicates the equipment holding / occupancy information, used to distinguish different processing stages (e.g., headphones not powered on, powering on, testing completed, etc.). To estimate the remaining time of the equipment task or the current work progress, normalization is performed using the maximum processing cycle. This refers to the historical utilization rate or occupancy statistics of equipment, used to quantify the level of resource busyness.

[0029] Depending on the equipment, the node status of the robotic arm is represented as follows: (2) in, This is the index of the robotic arm's current position within the set of reachable points (reachable points include: the conveyor belt for the test tray M0, the test tray M1, the start position M2, the detection box M3, the detection box M4, the detection box M5, the detection box M6, the good product test tray M7, and the material conveyor belt on the defective product side M8). This indicates a One-Hot encoding operation.

[0030] The suction cup node state is defined as follows: (3) in, This indicates the suction cup status (including idle and busy states), reflecting the working status of the suction cup.

[0031] The gripper node state is defined as follows: (4) To ensure the agent maintains the adaptability and continuity of the scheduling strategy even in the event of gripper malfunction, this scheme incorporates gripper malfunction modes into the gripper node state representation, enabling the agent to identify gripper malfunctions in real time and automatically adjust operation selections. It indicates the operating status of the gripper (including three states: idle, busy, and fault). This indicates the processing stage attributes of the clamped earphone (including four states: no material, not powered on, powered on, and detected). This indicates the acoustic test results of the headphones (including three states: no test material, good material, and defective material).

[0032] The states of the conveyor belt M0 and the node M1 of the test tray are as follows: (5) (6) in, This indicates the operating status of the conveyor belt of the material tray to be tested (including idle and busy states). This indicates the operating status of the material tray under test (including idle and busy states). This indicates the percentage of the task that has been completed in this material tray. Indicates the utilization rate of the material tray. , This indicates that there are still untested headphones on the test tray and that the headphones have accumulated time on the tray conveyor belt. This indicates the total operating time of the production line.

[0033] The node state vectors of power-on position M2 and detection boxes M3-M6 are respectively represented as follows: (7) (8) Similarly, to ensure the agent maintains the adaptability and continuity of the scheduling strategy even in the event of malfunctions in the power-on position and the detection box, this solution incorporates the fault mode into the state representation of the aforementioned nodes. This allows the agent to identify malfunctions in the power-on position and the detection box in real time and automatically adjust its operation selection. Furthermore, considering the fluctuations in processing time of the power-on position and the detection box, to enable the agent to perceive and dynamically assess the remaining processing time of the current process, this invention incorporates the expected processing time of both types of nodes and their normalized representation into the state features, thereby supporting adaptive optimization of the scheduling strategy under uncertain processing delays. It indicates the operating status of the power-on position (including four states: idle, busy, completed, and fault). This represents the normalized expected processing time for the start-up position. , This represents the historical number of processing operations performed at the start-up position. Indicates the boot position number Secondary historical processing time, This represents the average processing time at each machine station. This indicates the utilization rate of the operating position. , This indicates the total time taken for the power-on unit to perform the power-on operation. Indicates the testing box The operating status (including two states: on, off, and fault). Indicates the testing box Is there any material inside? This represents the normalized expected processing time. , For historical processing times, Indicates the testing box No. Secondary historical processing time, Indicates the testing box The average processing time, Indicates the testing box Work utilization rate , Indicates the testing box The total time spent performing the detection operation. Finally, the states of all device nodes are combined to form a complete reinforcement learning input space: (9) Through the structured state representation described above, the agent is equipped with global state perception capabilities, providing an accurate and scalable input foundation for subsequent device interaction modeling and scheduling strategy decision-making.

[0034] Next, to fully reflect the heterogeneous attributes of various equipment in the production line, this invention adopts an independent embedding method for different types of equipment nodes. This allows the embedded network to perform differentiated encoding and expression based on the unique functional characteristics of each equipment, thereby achieving effective decoupling and enhancement of node semantics. Through this method, nodes such as robotic arms, grippers, start-up positions, and inspection boxes can accurately express their positional capabilities, holding status, processing progress, and inspection task attributes, avoiding information aliasing caused by uniform encoding. This enables the model to more accurately perceive the weight differences of equipment in scheduling decisions. Therefore, this invention... Using independent fully connected layers, the states of the device node set are mapped to a feature space of uniform dimension, thereby obtaining node embedding vectors. Its calculation can be expressed as: (10) in, Indicates a node The fully connected layer that performs the embedding operation. This is the learnable weight matrix corresponding to this node. For nodes State feature dimension For nodes Embedding dimension This indicates the modification of the linear unit activation function. This step ensures that heterogeneous devices (grippers, power-on positions, detection boxes, etc.) are mapped to the same feature space for subsequent interactive computation.

[0035] Significant task dependencies exist between devices in an earphone acoustic testing production line; the operational status of one device often directly affects the schedulability of subsequent devices. To enable the agent to accurately understand this dynamic coupling, this invention employs a learnable attention mechanism to calculate the interaction relationships between nodes. This allows the model to consider both the state of individual nodes and the resource associations between different devices when making decisions, achieving adaptive modeling of device relationships. First, the embedded feature vectors of all nodes are concatenated sequentially to form a node feature matrix: (11) in, Represents the node feature matrix, This represents the number of device nodes. Subsequently, a learnable linear transformation matrix is ​​used. Mapping node features yields the transformed node representation: (12) For any two nodes Its attention relevance score is defined as: (13) in, Representing the node feature matrix The OK, Representing the node feature matrix The List, This represents a vector concatenation operation. A learnable weight vector used to measure the nodes For nodes Information attention level This represents a leaky linear rectified function. To obtain normalized directional attention weights, softmax is applied to all associated edges of each node: (14) in, Represents nodes The set of neighboring nodes with feasible task dependencies. This ultimately forms an adaptive directed adjacency matrix: (15) Elements in the matrix Indicates from device To the equipment The directed interaction strength. To accurately characterize the dependencies and collaborations between different devices on the headphone acoustic testing production line, this invention abstracts the production line structure into a directed graph, where the device features... As a set of nodes in a directed graph, the adaptive directed adjacency matrix As a set of edges in a directed graph, this invention then utilizes the message passing mechanism of a graph neural network to achieve information interaction and feature updates between device nodes. This allows each node to comprehensively consider its own state and the operating characteristics of its upstream and downstream devices, thereby extracting a high-level semantic representation reflecting the global dynamics of the production line. For the first... The update formula for layer graph convolution calculation is expressed as follows: (16) in, Indicates the first Layer node feature matrix It is a learnable linear transformation matrix. It provides the direction and weight of information propagation between nodes. The above formula represents the nodes. The features are formed by a weighted aggregation of its own features and the features of its neighboring nodes with which it has directed connections. Because It is directional, and information propagation follows bidirectional dependencies between devices, enabling the model to capture global device interaction characteristics during the update process. After multiple layers of message passing, the final node representation is obtained: (17) This matrix contains the contextual features and interaction information of each device within the global production environment. Subsequently, the aggregated features from different workstations are concatenated, and a one-dimensional convolutional neural network is used to output the agent's action probability distribution. The sub-action corresponding to the maximum Q-value is then selected as the optimal decision for the current state. (18) in, This indicates a splicing operation. This represents a one-dimensional convolution operation. Indicates the current state Next action The probability is used to guide the agent in making scheduling decisions at different production stages, such as "retrieving the earphone from the test tray", "placing the unpowered earphone in the power-on position", "transferring the powered-on earphone to the testing box", and "placing the appropriate tested tray according to the earphone testing results". This mechanism enables the agent to make adaptive decisions under conditions of processing time fluctuations and equipment failures, thereby maintaining the efficient operation of the production line.

[0036] To ensure that the agent learns efficient strategies that conform to production line inspection constraints during training, this invention designs a composite reward function to guide the agent to achieve an optimal trade-off between time efficiency, equipment utilization, and balance. At each time step... The agent, based on the current state Execute action Timely environmental return reward The reward function of this invention is defined as follows: (19) in, This is a time efficiency term used to measure the time cost of executing an action, encouraging agents to complete processing in shorter cycles. Its specific calculation method is as follows: (20) in, and These represent the sub-actions in the motion space: "The suction cup picks up the test tray from the conveyor belt M0" and "The suction cup places the test tray M1 into the conveyor belt of the good product test tray".

[0037] To incentivize high throughput, a reward is given when the inspection process completes two trays of materials. The reward is inversely proportional to the processing time. The specific calculation method is as follows: (twenty one) in, Based on the basic reward constant, and These are the start and end times for completing the two material tray testing tasks, respectively.

[0038] As a waiting penalty, a time penalty is applied when an agent's action causes the system to wait (e.g., arriving at a workstation before completion of processing). (twenty two) in, This indicates the duration of invalid waits in the system.

[0039] This is an idle workstation penalty item, used to prevent equipment from running idle or testing boxes from being used for extended periods. (twenty three) in, Indicates the current total time. This indicates the time it took to complete the inspection of the previous test tray. Indicates device The expected completion time is given, and the penalty increases with the square of the deviation when inspection delays or timeouts occur.

[0040] This is a device utilization balancing term, used to guide the scheduling strategy to achieve load balancing among multiple detection boxes. (twenty four) (25) (26) in, The average utilization rate of the testing box, To address the differences in test chamber utilization, this measure encourages load balancing of test chambers, thereby improving the overall resource utilization efficiency of the production line.

[0041] Finally, to prevent the agent from exhibiting potentially destructive behavior or ineffective strategies, set... This is a penalty for illegal operations. If an action violates physical or logical constraints (such as incorrect placement or mismatched equipment status), a stronger penalty will be immediately applied, and the current task will be terminated. (27) To further improve the training stability and convergence speed of the agent, this invention employs a Prioritized Experience Replay (PER) training strategy. The PER mechanism introduces a sampling mechanism based on temporal-difference (TD) error. Let the samples... The TD error is: (28) in, For instant rewards, As a discount factor, and These represent the current state and the action, respectively. Indicates the next state. and These are the parameters for the online network and the target network, respectively. TD error. The larger the value, the greater the improvement value of the sample for the current strategy. Based on this, the sampling probability of an empirical sample is defined as: (29) in, Indicates sample priority, It is a smoothing constant. Used to adjust the effect of priority on the sampling distribution. When When sampling degenerates into uniform randomness; when At this time, sampling tends to favor samples with high TD errors. Furthermore, since non-uniform sampling introduces estimation bias, an importance-sampling weight is needed to correct for this bias and ensure unbiased training. (30) in, The total number of samples in the replay pool. The strength of the correction is controlled, typically increasing gradually during training to ensure convergence in later stages. Finally, during gradient update, the strength of the correction is adjusted for each sample. Weighting the loss function: (31) Through the above mechanisms, experience-first replay can utilize key samples that contribute more to learning more frequently during training, thereby effectively accelerating convergence, improving training efficiency, and enhancing the stability of the agent under complex tasks.

[0042] In this embodiment of the invention, the reinforcement learning-based detection production line scheduling method based on graph neural networks includes: acquiring the node state variables of equipment nodes and materials; combining the node state variables of the equipment nodes and materials into a set of equipment nodes, and using an independent fully connected layer to map the state of the set of equipment nodes to a feature space of a unified dimension, thereby obtaining node embedding vectors; concatenating the embedding feature vectors of all nodes in sequence to form a node feature matrix; transforming the node feature matrix to obtain an adaptive directed adjacency matrix; obtaining the final node through the node feature matrix, the adaptive directed adjacency matrix, and a learnable linear transformation matrix; selecting the sub-action corresponding to the maximum Q value as the optimal decision in the current state; performing state transition on the current equipment state according to the selected sub-action to obtain the node state variable at the next moment, iterating cyclically until the scheduling strategy of the detection production line is output; and performing production scheduling on the detection production line according to the scheduling strategy. By modeling the headphone acoustic detection production line as a graph structure with dynamic directed interaction relationships, the graph neural network is used to extract the global correlation features between each equipment node, and a deep reinforcement learning algorithm is combined to realize the autonomous learning and optimization decision of the intelligent scheduling strategy. By independently embedding different device nodes, the model can fully express the heterogeneous attributes between devices; by modeling the directed interaction relationships between devices through the attention mechanism, the agent can dynamically capture the dependencies between production line devices; through multi-layer graph feature aggregation and composite reward function optimization, the system can achieve stable and efficient scheduling decisions under non-ideal working conditions such as processing time jitter, equipment abnormalities and complex task switching, thereby significantly improving the overall detection efficiency and robustness of the acoustic inspection production line.

[0043] The core of this invention lies in the first-ever abstraction of the headphone acoustic testing production line into a directed graph structure. Different device nodes are modeled as graph nodes, and the interaction strength between nodes is adaptively learned using an attention mechanism, thereby accurately expressing the dynamic dependencies and task transmission directions between devices.

[0044] To address the heterogeneous attributes of different types of equipment in a production line, this invention employs an independent embedded network for differentiated feature mapping of each equipment node. This allows the embedded features to retain the individual semantics of the equipment while possessing a unified representation space, thus solving the problems of information aliasing and insufficient expression caused by traditional unified coding methods.

[0045] By introducing a self-attention mechanism, the agent can adaptively learn the directed interaction weights between devices during training, dynamically construct an adjacency matrix, and fully reflect the dependencies between various devices in the production line.

[0046] This invention sets up multiple composite rewards, including time efficiency rewards, task completion rewards, waiting penalties, idle penalties, and equipment utilization balance rewards, and combines them with an illegal operation penalty mechanism, enabling the intelligent agent to adaptively balance production efficiency, system stability, and resource balance under multi-objective constraints, and achieve optimal scheduling strategy learning.

[0047] This invention introduces random disturbances such as equipment failures and processing time fluctuations into simulation training, enabling the model to learn robust and adaptive scheduling strategies, ensuring that high production capacity and continuous system operation can still be maintained even when some equipment is abnormal or processing is delayed.

[0048] Compared with traditional manually pre-set scheduling strategies and general reinforcement learning methods, this invention has the following significant advantages: This invention achieves autonomous learning and continuous optimization of scheduling strategies through deep reinforcement learning, enabling dynamic adjustment of execution actions based on the real-time status of the production line, eliminating reliance on human experience and realizing fully automated decision-making.

[0049] By introducing graph neural network structure modeling and node independent embedding mechanism, the agent can capture the complex task dependencies and resource coupling relationships between devices, and fully understand the global operating status of the production line, thereby improving the accuracy of strategy reasoning.

[0050] This invention trains the learned strategy by introducing disturbances such as equipment failure and time jitter in a virtual simulation environment, thereby enhancing its robustness and generalization in practical applications. Furthermore, the model structure is flexibly adaptable to different numbers or types of device nodes, exhibiting excellent scalability.

[0051] By optimizing the multidimensional composite reward mechanism and co-modeling the directed attention graph structure, this invention achieves high parallel scheduling and resource load balancing of the inspection process. Under the same hardware configuration, it can significantly improve the inspection efficiency of the production line and reduce the invalid waiting time.

[0052] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0053] Reference Figure 4 This diagram illustrates a structural block diagram of an embodiment of a reinforcement learning-based production line scheduling device for graph neural networks, which may specifically include the following modules: The acquisition module 301 is used to acquire the node status quantities of equipment nodes and materials; The node embedding vector module 302 is used to combine the node state quantities of the device nodes and materials into a device node set, and use an independent fully connected layer to map the state of the device node set to a feature space of a unified dimension, thereby obtaining a node embedding vector. The node feature matrix module 303 is used to concatenate the embedded feature vectors of all nodes in sequence to form a node feature matrix; The directed adjacency matrix module 304 is used to obtain an adaptive directed adjacency matrix after transforming the node feature matrix. The final node module 305 is used to obtain the final node through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; The optimal decision module 306 is used to select the sub-action corresponding to the maximum Q value as the optimal decision in the current state; The production scheduling module 307 is used to perform state transition on the current equipment state according to the selected sub-action, obtain the node state quantity at the next moment, iterate in a loop until the scheduling strategy of the detection production line is output, and perform production scheduling on the detection production line according to the scheduling strategy.

[0054] Preferably, the device includes an intelligent agent that performs actions based on the current state, and the environment returns a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

[0055] Preferably, the device further includes: The parameter acquisition module is used to acquire instant rewards, discount factors, current status, current action, next status, online network parameters, and target network parameters. The time difference error module is used to obtain the time difference error based on the instant reward, discount factor, current state, current action, next state, online network parameters, and target network parameters.

[0056] Preferably, the node status quantities of the equipment nodes and materials include the node status quantities corresponding to the robotic arm, gripper, start-up position, and detection box.

[0057] Preferably, the directed adjacency matrix module includes: The mapping submodule is used to map the node feature matrix using a learnable linear transformation matrix to obtain the transformed node. The correlation molecular module is used to calculate the attention correlation score between any two nodes. It performs softmax processing on all associated edges of each node to form an adaptive directed adjacency matrix.

[0058] Preferably, the device further includes: The first determining module is used to determine the device features as a set of nodes in a directed graph; The second determining module is used to determine the adaptive directed adjacency matrix as the edge set of the directed graph.

[0059] The modules in the aforementioned reinforcement learning-based production line scheduling device based on graph neural networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0060] The graph neural network-based reinforcement learning detection production line scheduling device provided above can be used to execute the graph neural network-based reinforcement learning detection production line scheduling method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0061] In one embodiment, a computer device is provided, which may be an automotive electronic instrument panel device, and its internal structure diagram may be as follows: Figure 5As shown, the automotive electronic instrument panel includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a reinforcement learning-based production line scheduling method based on graph neural networks. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0062] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0063] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Obtain the node status of equipment nodes and materials; The node state variables of the equipment nodes and materials are combined into a set of equipment nodes. An independent fully connected layer is used to map the state of the set of equipment nodes to a feature space of a unified dimension, thereby obtaining the node embedding vector. The embedded feature vectors of all nodes are concatenated in order to form a node feature matrix; After transforming the node feature matrix, an adaptive directed adjacency matrix is ​​obtained; The final node is obtained through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; Select the sub-action corresponding to the maximum Q value as the optimal decision in the current state; The current device state is transitioned according to the selected sub-action to obtain the node state quantity at the next moment. This process is repeated until the scheduling strategy of the detection production line is output. The production of the detection production line is then scheduled according to the scheduling strategy.

[0064] Preferably, the method includes an agent performing an action based on the current state, and the environment returning a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

[0065] Preferably, the method further includes: Obtain instant rewards, discount factors, current status, current action, next status, online network parameters, and target network parameters; The time difference error is obtained based on the instant reward, discount factor, current state, current action, next state, online network parameters, and target network parameters.

[0066] Preferably, the node status quantities of the equipment nodes and materials include the node status quantities corresponding to the robotic arm, gripper, start-up position, and detection box.

[0067] Preferably, the step of transforming the node feature matrix to obtain the adaptive directed adjacency matrix includes: The node feature matrix is ​​mapped by a learnable linear transformation matrix to obtain the transformed node; Calculate the attention relevance score between any two nodes, and perform softmax processing on all associated edges of each node to form an adaptive directed adjacency matrix.

[0068] Preferably, the method further includes: The device characteristics are defined as a set of nodes in a directed graph; The adaptive directed adjacency matrix is ​​defined as the set of edges of the directed graph.

[0069] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the node status of equipment nodes and materials; The node state variables of the equipment nodes and materials are combined into a set of equipment nodes. An independent fully connected layer is used to map the state of the set of equipment nodes to a feature space of a unified dimension, thereby obtaining the node embedding vector. The embedded feature vectors of all nodes are concatenated in order to form a node feature matrix; After transforming the node feature matrix, an adaptive directed adjacency matrix is ​​obtained; The final node is obtained through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; Select the sub-action corresponding to the maximum Q value as the optimal decision in the current state; The current device state is transitioned according to the selected sub-action to obtain the node state quantity at the next moment. This process is repeated until the scheduling strategy of the detection production line is output. The production of the detection production line is then scheduled according to the scheduling strategy.

[0070] Preferably, the method includes an agent performing an action based on the current state, and the environment returning a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

[0071] Preferably, the method further includes: Obtain instant rewards, discount factors, current status, current action, next status, online network parameters, and target network parameters; The time difference error is obtained based on the instant reward, discount factor, current state, current action, next state, online network parameters, and target network parameters.

[0072] Preferably, the node status quantities of the equipment nodes and materials include the node status quantities corresponding to the robotic arm, gripper, start-up position, and detection box.

[0073] Preferably, the step of transforming the node feature matrix to obtain the adaptive directed adjacency matrix includes: The node feature matrix is ​​mapped by a learnable linear transformation matrix to obtain the transformed node; Calculate the attention relevance score between any two nodes, and perform softmax processing on all associated edges of each node to form an adaptive directed adjacency matrix.

[0074] Preferably, the method further includes: The device characteristics are defined as a set of nodes in a directed graph; The adaptive directed adjacency matrix is ​​defined as the set of edges of the directed graph.

[0075] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0076] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0081] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0082] The foregoing has provided a detailed description of a reinforcement learning detection production line scheduling method based on graph neural networks, a reinforcement learning detection production line scheduling device based on graph neural networks, a computer device, and a storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A reinforcement learning-based production line scheduling detection method based on graph neural networks, characterized in that, The method includes: Obtain the node status of equipment nodes and materials; The node state variables of the equipment nodes and materials are combined into a set of equipment nodes. An independent fully connected layer is used to map the state of the set of equipment nodes to a feature space of a unified dimension, thereby obtaining the node embedding vector. The embedded feature vectors of all nodes are concatenated in order to form a node feature matrix; After transforming the node feature matrix, an adaptive directed adjacency matrix is ​​obtained; The final node is obtained through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; Select the sub-action corresponding to the maximum Q value as the optimal decision in the current state; The current device state is transitioned according to the selected sub-action to obtain the node state quantity at the next moment. This process is repeated until the scheduling strategy of the detection production line is output. The production of the detection production line is then scheduled according to the scheduling strategy.

2. The reinforcement learning-based production line scheduling method based on graph neural networks according to claim 1, characterized in that, The method includes an agent performing an action based on the current state, and the environment returning a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

3. The reinforcement learning-based production line scheduling method based on graph neural networks according to claim 1, characterized in that, The method further includes: Obtain instant rewards, discount factors, current status, current action, next status, online network parameters, and target network parameters; The time difference error is obtained based on the instant reward, discount factor, current state, current action, next state, online network parameters, and target network parameters.

4. The reinforcement learning-based production line scheduling method based on graph neural networks according to claim 1, characterized in that, The node status quantities of the equipment nodes and materials include the node status quantities corresponding to the robotic arm, gripper, start position, and detection box.

5. The reinforcement learning-based production line scheduling method based on graph neural networks according to claim 1, characterized in that, The process of transforming the node feature matrix to obtain the adaptive directed adjacency matrix includes: The node feature matrix is ​​mapped by a learnable linear transformation matrix to obtain the transformed node; Calculate the attention relevance score between any two nodes, and perform softmax processing on all associated edges of each node to form an adaptive directed adjacency matrix.

6. The reinforcement learning-based production line scheduling method based on graph neural networks according to claim 1, characterized in that, The method further includes: The device characteristics are defined as a set of nodes in a directed graph; The adaptive directed adjacency matrix is ​​defined as the set of edges of the directed graph.

7. A reinforcement learning-based production line scheduling device for detection using graph neural networks, characterized in that, The device includes: The acquisition module is used to acquire the node status quantities of equipment nodes and materials; The node embedding vector module is used to combine the node state variables of the device nodes and materials into a device node set, and use an independent fully connected layer to map the state of the device node set to a feature space of a unified dimension, thereby obtaining the node embedding vector. The node feature matrix module is used to concatenate the embedded feature vectors of all nodes in sequence to form a node feature matrix; The directed adjacency matrix module is used to obtain an adaptive directed adjacency matrix after transforming the node feature matrix. The final node module is used to obtain the final node through the node feature matrix, the adaptive directed adjacency matrix, and the learnable linear transformation matrix; The optimal decision module is used to select the sub-action corresponding to the maximum Q value as the optimal decision in the current state. The production scheduling module is used to perform state transitions on the current equipment state based on the selected sub-actions, obtain the node state quantity at the next moment, and iterate in a loop until the scheduling strategy of the inspection production line is output, and to perform production scheduling on the inspection production line according to the scheduling strategy.

8. The reinforcement learning-based production line scheduling device based on graph neural networks according to claim 7, characterized in that, The device includes an intelligent agent that performs actions based on the current state, and the environment returns a reward; the reward includes a task completion reward, a waiting penalty, an idle workstation penalty, an equipment utilization balancing penalty, and an illegal operation penalty.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning detection production line scheduling method based on graph neural networks as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning detection production line scheduling method based on graph neural networks as described in any one of claims 1 to 6.

Images