A method and system for adjusting a human-robot collaborative assembly station based on graph learning

Abstract

The application relates to the technical field of mechanical equipment, and discloses an adjustment method and system of a man-machine cooperation assembly station based on graph learning. The method comprises the following steps: constructing an integrated virtual reconstruction model based on a training method; inputting production change information into the virtual reconstruction model to output an adjustment assembly strategy. The training method comprises the following steps: acquiring an image set of an assembly station and pre-processing the image set; establishing a graph structure based on the assembly station according to the pre-processed image set, and establishing a graph convolution network according to the graph structure; optimizing the graph convolution network to obtain an optimized graph convolution network; training the optimized graph convolution network, defining a loss function to adjust the optimized graph convolution network in the training process, and obtaining a trained graph convolution network. Through the form of simulation of the man-machine cooperation assembly system, virtual-real synchronization and fusion, the cost of the man-machine cooperation assembly station is greatly reduced, and the production efficiency of an enterprise is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical equipment, and in particular to an adjustment method and system for a human-machine collaborative assembly station based on graph learning. Background Art

[0002] Human-robot collaborative assembly stations are becoming increasingly popular in many fields, but they also face numerous challenges. On the one hand, internal process improvements require significant manpower, material, and financial resources, and must meet market demands as quickly as possible. On the other hand, market demand is constantly changing, requiring companies to dynamically adjust production plans and necessitating greater flexibility in assembly systems. Furthermore, customer needs are becoming increasingly diverse and personalized, making customization an inevitable trend. First, existing human-robot collaborative assembly stations are slow to respond. Internal process improvements require time and resources for employee training, and machines require time to adapt. This results in a significant short-term decline in productivity and an inability to quickly respond to market demands. Second, existing human-robot collaborative assembly stations are costly. Failure to find an optimal solution in the short term, whether due to process improvements or external interference, will create significant financial pressure. Finally, existing human-robot collaborative assembly stations lack flexibility, significantly reducing efficiency when faced with both implicit interference and explicit changes.

[0003] The prior art discloses a human-machine collaborative assembly workstation and method. The human-machine collaborative assembly workstation includes a conveyor roller, a collaborative robot, an assembly locating sleeve, a stamping mechanism, and a control system. The control system includes a controller and a human-machine interface. A tooling plate is provided on the conveyor roller, which supports a gear shaft and bearings. The conveyor roller drives the tooling plate, thereby driving the gear shaft and bearings. The collaborative robot can move the gear shaft between the assembly locating sleeve and the stamping mechanism. The stamping mechanism is used to stamp the gear shaft with the bearing. The controller is electrically connected to the human-machine interface, the conveyor roller, the collaborative robot, and the stamping mechanism. This prior art improves the efficiency of the bearing press-fitting process by using a collaborative robot and a human operator. Furthermore, the collaborative robot is used to place the gear shaft with the bearing into the stamping mechanism, improving the safety of the assembly process. However, this prior art suffers from poor flexibility and cannot respond quickly to process improvements or changes in market demand, resulting in a significant decrease in production efficiency. Training professional operators also requires a significant amount of time and financial costs. Summary of the Invention

[0004] The primary objective of this invention is to overcome the problems of the existing technology by providing a graph-learning-based human-machine collaborative assembly method. This method significantly reduces the cost of human-machine collaborative assembly stations through the synchronization and integration of virtual and real systems through the simulation of human-machine collaborative assembly systems. Existing technologies require significant time, space, and capital to adjust and adapt to process changes, environmental changes, and other challenges. This method, however, reconstructs the model, simulates adjustments and optimizations, and optimizes production plans offline, shortening the integration cycle and improving production efficiency.

[0005] As another object of the present invention, a system adapted to the method according to the aforementioned object is also provided.

[0006] As another object of the present invention, a non-volatile storage medium suitable for storing a computer program implemented according to the method described is provided.

[0007] To achieve the above objectives, the present invention provides a method for adjusting a human-machine collaborative assembly station based on graph learning. The adjustment method is based on a training method for a graph convolutional network for an assembly station, and is characterized in that the training method includes:

[0008] Acquiring an image set of an assembly station and preprocessing the image set;

[0009] A graph structure based on the assembly station is established according to the preprocessed image set, and a graph convolutional network is established according to the graph structure, specifically including: obtaining an adjacency matrix and a node feature matrix based on the graph structure, normalizing the adjacency matrix to obtain a normalized adjacency matrix; constructing a graph convolutional network according to the normalized adjacency matrix and the node feature matrix; and the method for constructing the graph convolutional network is as follows:

[0010]

[0011] in For the The node feature matrix of the layer, is the normalized adjacency matrix, For the The weight matrix of the layer, is the activation function, where the normalized adjacency matrix is calculated as follows:

[0012]

[0013] in, is the degree matrix, is the identity matrix, is the adjacency matrix;

[0014] Optimizing the graph convolutional network to obtain an optimized graph convolutional network;

[0015] Train the optimized graph convolutional network, define a loss function during the training process to adjust the optimized graph convolutional network, and obtain a trained graph convolutional network, wherein the loss function includes a first loss function and a second loss function, wherein the first loss function is used to evaluate the performance of the loss-optimized graph convolutional network in predicting the attributes or values ​​of the nodes, and the second loss function is used to evaluate the performance of the loss-optimized graph convolutional network in predicting whether there is an edge between two nodes, wherein the first loss function is as follows:

[0016]

[0017] in, is the number of nodes in the graph structure based on the target assembly station, is the true value of the node, is the predicted value of the node;

[0018] The second loss function is as follows:

[0019]

[0020] in, is the element of the real adjacency matrix, representing the node and Does the edge between them exist? For the optimized graph convolutional network nodes and The predicted probability that an edge exists between them.

[0021] An integrated virtual reconstruction model is constructed based on the training method; production change information is input into the virtual reconstruction model, and an adjusted assembly strategy is output.

[0022] Furthermore, the preprocessing includes removing missing or abnormal images in the image set and unifying the formats of the remaining images in the image set to obtain a preprocessed image set.

[0023] Furthermore, the optimization of the graph convolutional network specifically includes: optimizing the parameters of the graph convolutional network to obtain a first graph convolutional network, wherein the parameters include convolution batch size, convolution kernel size and number of convolution layers; in the first graph convolutional network, using the graph attention pooling method instead of the global average pooling method to calculate the node feature weights to obtain an optimized graph convolutional network.

[0024] Furthermore, the integrated virtual reconstruction model specifically includes: a visual sensor, a soft PLC and its external modules, and simulation software;

[0025] The visual sensor is used to collect an image set of the target assembly station;

[0026] The soft PLC and its external modules train a graph convolutional network for the target assembly station based on the training method; obtain the trained graph convolutional network for the target assembly station, and the trained graph convolutional network for the target assembly station receives the image set of the target assembly station to obtain optimization data;

[0027] The simulation software obtains an optimization model according to the optimization data, and obtains an adjustment assembly strategy based on the optimization model and the original model.

[0028] To achieve another object of the present invention, the present invention further provides a human-machine collaborative assembly system based on graph learning. The system is based on the human-machine collaborative assembly method based on graph learning, and includes:

[0029] Building module: building an integrated virtual reconstruction model based on the training method;

[0030] Output module: inputs the production change information into the integrated virtual reconstruction model and outputs the adjusted assembly strategy.

[0031] In order to achieve another purpose of the present invention, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the method for adjusting a human-machine collaborative assembly station based on graph learning is implemented.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention uses graph convolutional neural networks to analyze and learn nodes and establish an integrated virtual reconstruction model. This model can be simulated in human-machine assembly collaboration systems and simulation systems to perform iterative optimization of new process production design and execution, effectively improving the rationality of the entire system design and realizing intelligent operation and maintenance of the human-machine assembly collaboration system.

[0034] 2. Through the simulation of human-machine collaborative assembly systems, virtual and real synchronization and integration are achieved, significantly reducing the cost of human-machine collaborative assembly stations. Existing technologies require significant time, space, and capital to adjust and adapt to process changes or environmental changes. However, the method of this invention trains and reconstructs models, simulates and optimizes adjustments, and thus optimizes production plans offline, shortening the joint debugging and testing cycle and improving enterprise production efficiency.

[0035] 3. By collecting motion and production data from collaborative human-robot assembly stations, and using image data that can be used to document the integrity of physical equipment, this data, when integrated with the assembly execution system, enables lifecycle quality control and anti-counterfeiting measures, thereby tracking the lifecycle of assembled parts. This significantly reduces the cost of manual verification and workpiece tracking, providing technical support for transparent monitoring of collaborative human-robot assembly. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a method for adjusting a human-machine collaborative assembly station based on graph learning according to an embodiment of the present invention;

[0037] Figure 2 This is a block diagram of an adjustment system for a human-machine collaborative assembly station based on graph learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0039] In the description of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of the present invention and simplify the description. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0041] Furthermore, in the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0042] Example 1

[0043] like Figure 1 As shown, a method for adjusting a human-machine collaborative assembly station based on graph learning in a preferred embodiment of the present invention includes:

[0044] S1: Obtain the image set of the assembly station and preprocess the image set;

[0045] In a feasible embodiment, the preprocessing includes removing missing or abnormal images in the image set and unifying the formats of the remaining images in the image set to facilitate subsequent processing, thereby obtaining a preprocessed image set.

[0046] S2: Establish a graph structure based on the assembly station according to the preprocessed image set, and establish a graph convolutional network based on the graph structure;

[0047] In a feasible embodiment, S2 specifically includes:

[0048] S2.1: Based on the preprocessed image set, a graph structure based on the assembly station is established. Specifically, (1) nodes and edges are determined. The elements of each assembly station production line (such as machines, tools, workpieces, etc.) can be represented as nodes in the graph, and the relationships between nodes (such as workflow, tool usage, etc.) can be represented by edges; (2) node and edge attributes are defined. Adding attributes to each node and edge can describe the elements of the assembly station in more detail. For machines, its node attributes include type, status, and production capacity; for workpieces, its node attributes include size, weight, and material. Similarly, edges can also contain attributes to represent the characteristics of the relationship. For example, in a certain workflow, the attributes can be the required time, priority, etc.; (3) Build a graph structure and use graph data structures (such as adjacency list: use a list to store each node and its adjacent nodes. Adjacency matrix: use a two-dimensional array to represent the connection relationship between nodes. The rows and columns of the matrix correspond to the node information in the graph, and the value of the element represents the existence or weight of the edge.) to realize the above nodes and edges.

[0049] S2.2: Based on the graph structure, obtain an adjacency matrix and a node feature matrix, normalize the adjacency matrix, and obtain a normalized adjacency matrix; construct an adjacency matrix according to the graph structure. And the node feature matrix, the adjacency matrix A is normalized and recorded as , normalized expression:

[0050]

[0051] in, is the degree matrix, is the identity matrix;

[0052] S2.3: Construct a graph convolutional network based on the normalized adjacency matrix and the node feature matrix. The construction method is as follows:

[0053]

[0054] in For the The node feature matrix of the layer, is the normalized adjacency matrix, For the The weight matrix of the layer, is the activation function.

[0055] S3: Optimizing the graph convolutional network to obtain an optimized graph convolutional network;

[0056] In a feasible embodiment, S3 specifically includes:

[0057] S3.1: Optimize the parameters of the graph convolutional network to obtain the first graph convolutional network. These parameters include the convolution batch size, convolution kernel size, and number of convolution layers. Specifically, experiment with different kernel parameter settings (such as the stride, padding, and kernel size of the convolution layer) in a distributed environment. Three parameters are considered: the convolution batch size, which primarily refers to the number of samples used to calculate the gradient at each iteration. Experiment with different batch sizes, such as 32, 64, and 128. Smaller batch sizes can improve model generalization but may increase training time. The convolution kernel size, which primarily refers to the size of the filter used in the convolution operation, is selected based on the characteristics of the input data, such as 3x3 or 5x5. The number of convolution layers, which primarily refers to the number of convolution layers in the network, is determined. The impact of increasing or decreasing convolution layers on model performance is observed. Analyze how kernel parameters are transferred and updated between different compute nodes. Different network topologies were set to observe the effects, performance, and efficiency of parameter transfer, thereby analyzing the impact of different kernel parameters on the transfer parameters. By comparing multiple sets of data, the optimal kernel parameter solution was found, and finally the first convolutional network was obtained.

[0058] S3.2: In the first graph convolutional network, the graph attention pooling method is used instead of the global average pooling method to calculate the node feature weights, thereby obtaining an optimized graph convolutional network. Specifically, considering the temporal factors, each assembly feature may change over time. The graph attention pooling method is used to dynamically calculate the weight of each feature before and after the assembly station, and the weighted sum of each feature is performed to obtain a comprehensive feature representation. Attention pooling can dynamically adjust the focus of features before and after the assembly station, thereby improving information utilization. It also reduces the learning of other parameters and helps improve the accuracy and robustness of the model by focusing on the most relevant feature information.

[0059] S4: Train the optimized graph convolutional network, define a loss function during the training process to adjust the optimized graph convolutional network, and obtain a trained graph convolutional network.

[0060] In a feasible embodiment, the loss function includes a first loss function and a second loss function, wherein the first loss function is used to evaluate the performance of the loss-optimized graph convolutional network in predicting the attribute or value of a node, and the second loss function is used to evaluate the performance of the loss-optimized graph convolutional network in predicting whether there is an edge between two nodes, wherein the first loss function is as follows:

[0061]

[0062] in, is the number of nodes in the graph structure based on the target assembly station, is the true value of the node, is the predicted value of the node;

[0063] The second loss function is as follows:

[0064]

[0065] in, is the element of the real adjacency matrix, representing the node and Does the edge between them exist? For the optimized graph convolutional network nodes and The predicted probability of the existence of an edge between them. Define the loss function, minimize the target value, and use backpropagation to adjust the parameters of the optimized graph convolutional network to obtain the trained graph convolutional network.

[0066] S5: Build an integrated virtual reconstruction model based on the training method described in S1-S4;

[0067] In a feasible embodiment, the integrated virtual reconstruction model specifically includes: a visual sensor, a soft PLC and its external modules, and simulation software;

[0068] The visual sensor is used to capture images of the target assembly station and transmit them to the software PLC and its external modules via the uplink and downlink channels of the C&C. A software PLC is a system based on a general-purpose computer platform that implements the control functions of a traditional hardware PLC (Programmable Logic Controller) through software. It utilizes computer resources such as the CPU, memory, and hard drive, along with specialized software tools, to simulate the operating principles of a PLC, thereby achieving automated control of industrial equipment.

[0069] The soft PLC and its external modules train a graph convolutional network for the target assembly station based on the training method; obtain the trained graph convolutional network for the target assembly station, the trained graph convolutional network for the target assembly station receives the image set of the target assembly station, obtains optimization data, and transmits the optimization data to the simulation software through the industrial network;

[0070] The simulation software obtains an optimization model based on the optimization data, and obtains an adjusted assembly strategy based on the optimization model and the original model. Each production change triggers the online optimization of the simulation model, the simulation model rewriting mechanism, modifies and updates the rule set, reconstructs the configuration diagram and the system controller, and sends the optimization instructions back to the PLC. The PLC then sends instructions from the simulation model translator to the lower-level device controller through the downlink channel, thereby adjusting the physical configuration of the assembly node and better assembling and packaging the workpiece. Similarly, each node production change will trigger the simulation model rewriting mechanism, reconstruct the reasoning based on the difference calculation before and after the data, and integrate the reconstructed optimization model. Finally, the global model is incrementally improved according to the latest state parameters of each node, and updated to the local policy database of each node, and then optimization decisions are made to achieve intelligent incremental accumulation and reconstruction mode improvement.

[0071] S6: Input the production change information into the integrated virtual reconstruction model and output the adjusted assembly strategy.

[0072] Input production change information, such as product switching, equipment adjustment, and resource allocation, into the integrated virtual reconstruction model to obtain the adjusted assembly strategy.

[0073] Example 2

[0074] like Figure 2 As shown, a human-machine collaborative assembly system based on graph learning according to an embodiment of the present invention includes:

[0075] Building module: building an integrated virtual reconstruction model based on the training method;

[0076] Output module: inputs the production change information into the integrated virtual reconstruction model and outputs the adjusted assembly strategy.

[0077] The system of this embodiment is based on the human-machine collaborative assembly method based on graph learning proposed in Example 1. Therefore, the options proposed in Example 1 are also applicable to this embodiment and will not be repeated here.

[0078] Example 3

[0079] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer-stored program is executed by a processor, the method for adjusting a human-machine collaborative assembly station based on graph learning is implemented.

[0080] The working process of the present invention is:

[0081] In summary, embodiments of the present invention provide a graph-based learning-based human-machine collaborative assembly method, system, and storage medium. This method uses a graph convolutional neural network to analyze and learn nodes, establishing an integrated virtual reconstruction model. This model can be simulated in both the human-machine collaborative assembly system and the simulation system, enabling iterative optimization of new process production design and execution. This effectively improves the rationality of the overall system design and enables intelligent operation and maintenance of the human-machine collaborative assembly system. Furthermore, through the simulation of the human-machine collaborative assembly system, virtual and real synchronization and integration are achieved, significantly reducing the cost of the human-machine collaborative assembly station. Existing technologies require significant time, space, and capital to adjust and adapt to process changes or environmental changes. However, the present invention utilizes a method to train a reconstruction model, simulate adjustments and optimizations, and optimize production plans offline, shortening the joint debugging and testing cycle and improving enterprise production efficiency. By collecting motion and production data from the human-machine collaborative assembly station and binding it to the assembly execution system based on image data that can be used to record the untampered physical equipment, it can achieve product lifecycle quality control and anti-counterfeiting, thereby tracking the lifecycle of the assembly parts. This significantly reduces the cost of manual verification and workpiece tracking, providing technical support for transparent monitoring of human-machine collaborative assembly.

[0082] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A method for adjusting a human-machine collaborative assembly station based on graph learning, the method being based on a training method for a graph convolutional network for an assembly station, characterized in that: Training methods include: Acquiring an image set of an assembly station and preprocessing the image set; A graph structure based on the assembly station is established according to the preprocessed image set, and a graph convolutional network is established according to the graph structure, specifically including: obtaining an adjacency matrix and a node feature matrix based on the graph structure, normalizing the adjacency matrix to obtain a normalized adjacency matrix; constructing a graph convolutional network according to the normalized adjacency matrix and the node feature matrix; and the method for constructing the graph convolutional network is as follows: in For the The node feature matrix of the layer, is the normalized adjacency matrix, For the The weight matrix of the layer, is the activation function, where the normalized adjacency matrix is calculated as follows: in, is the degree matrix, is the identity matrix, is the adjacency matrix; Optimizing the graph convolutional network to obtain an optimized graph convolutional network; Train the optimized graph convolutional network, define a loss function during the training process to adjust the optimized graph convolutional network, and obtain a trained graph convolutional network, wherein the loss function includes a first loss function and a second loss function, wherein the first loss function is used to evaluate the performance of the loss-optimized graph convolutional network in predicting the attributes or values ​​of the nodes, and the second loss function is used to evaluate the performance of the loss-optimized graph convolutional network in predicting whether there is an edge between two nodes, wherein the first loss function is as follows: in, is the number of nodes in the graph structure based on the target assembly station, is the true value of the node, is the predicted value of the node; The second loss function is as follows: in, is the element of the real adjacency matrix, representing the node and Does the edge between them exist? For the optimized graph convolutional network nodes and The predicted probability that there is an edge between An integrated virtual reconstruction model is constructed based on the training method; production change information is input into the virtual reconstruction model, and an adjusted assembly strategy is output.

2. The method for adjusting a human-machine collaborative assembly station based on graph learning according to claim 1, characterized in that: The preprocessing includes removing missing or abnormal images in the image set and unifying the formats of the remaining images in the image set to obtain a preprocessed image set.

3. The method for adjusting a human-machine collaborative assembly station based on graph learning according to claim 1, characterized in that: The optimizing the graph convolutional network specifically includes: optimizing the parameters of the graph convolutional network to obtain a first graph convolutional network, wherein the parameters include convolution batch size, convolution kernel size, and number of convolution layers; in the first graph convolutional network, using a graph attention pooling method instead of a global average pooling method to calculate node feature weights to obtain an optimized graph convolutional network.

4. The method for adjusting a human-machine collaborative assembly station based on graph learning according to claim 1, characterized in that: The integrated virtual reconstruction model specifically includes: a visual sensor, a soft PLC and its external modules, and simulation software; The visual sensor is used to collect an image set of the target assembly station; The soft PLC and its external modules train a graph convolutional network for the target assembly station based on the training method; obtain the trained graph convolutional network for the target assembly station, and the trained graph convolutional network for the target assembly station receives the image set of the target assembly station to obtain optimization data; The simulation software obtains an optimization model according to the optimization data, and obtains an adjustment assembly strategy based on the optimization model and the original model.

5. A human-machine collaborative assembly system based on graph learning, the system being based on the human-machine collaborative assembly method based on graph learning according to any one of claims 1 to 4, characterized in that: include: Building module: building an integrated virtual reconstruction model based on the training method; Output module: inputs the production change information into the integrated virtual reconstruction model and outputs the adjusted assembly strategy.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer stored program is executed by the processor, the method for adjusting a human-machine collaborative assembly station based on graph learning as described in any one of claims 1 to 4 is implemented.

Images