Human-machine cooperation assembly planning method based on real-time muscle fatigue of human body

Abstract

The invention provides a human-machine cooperation assembly planning method based on human body real-time muscle fatigue, and relates to the technical field of human-machine cooperation assembly manufacturing, and the method comprises the steps: constructing a muscle fatigue index model through a multi-angle motion image set, a human body skeletal muscle model and a biomechanical data set; training the deep learning model through the sample assembly action image set, and constructing an action category recognition model; according to the actual assembly action image set, the human skeletal muscle model, the muscle fatigue index model and the action category recognition model, the actual assembly time consumption and the actual fatigue accumulated value of the person are obtained through calculation; and based on the actual assembly time consumption of the robot, the actual assembly time consumption of the person and the actual fatigue accumulated value, an optimal population is obtained through calculation of a multi-target genetic algorithm, and an optimized man-machine cooperation assembly process is obtained through the optimal population. According to the method, human muscle fatigue and assembling action time consumption are monitored in a non-invasive and real-time mode, the man-machine cooperation assembling procedure is dynamically adjusted, the man-machine cooperation assembling efficiency is improved, human muscle fatigue is reduced, and meanwhile the method is more suitable for the actual assembling environment.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine collaborative assembly manufacturing, and in particular to a human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body. Background Art

[0002] With the rapid development of industry, human-robot collaboration is increasingly being used in industrial assembly. However, muscle fatigue caused by repetitive tasks not only affects assembly efficiency but can also cause long-term damage to workers' health. Existing research on dynamic planning of human-robot collaborative assembly sequences largely ignores the dynamic changes in human fatigue and its real-time impact on process planning, making it difficult to maximize the efficiency of human-robot collaboration in actual production.

[0003] Human muscle fatigue assessment is a technology that assesses fatigue levels by quantitatively analyzing changes in muscle function during sustained or repetitive activity. Accurate muscle fatigue assessment requires the integration of multidisciplinary knowledge, including biomechanics, exercise physiology, computer vision, and deep learning, to develop an analysis method based on motion capture, muscle force calculation, and mathematical models. This assessment technology can be applied to many fields requiring high concentration and prolonged work, such as human-robot collaboration, sports science, and industrial safety. Currently, research at home and abroad focuses on three types of assessment methods: physiological signals, behavioral characteristics, and task performance. Methods based on physiological signals (such as electromyography and electrocardiography) directly monitor muscle activity or autonomic nervous system responses through sensors, offering high accuracy but relying on wearable devices. Methods based on behavioral characteristics (such as joint angles and movement trajectories) utilize visual capture technology for non-contact assessment, but require high environmental adaptability. Methods based on task performance indirectly assess fatigue by analyzing work efficiency but are susceptible to interference from other factors. Despite significant progress in muscle fatigue assessment research, practical industrial deployment still faces challenges such as insufficient real-time performance and complex personalized modeling. Therefore, developing a high-precision, low-invasive real-time assessment method for muscle fatigue is of great significance for optimizing human-machine collaborative task allocation, protecting worker health and improving production efficiency.

[0004] In human-robot collaborative assembly, dynamic process planning is an important means of improving production efficiency and reducing worker fatigue. Dynamic task planning must not only consider the robot's efficient execution capabilities but also the human operator's performance and fatigue. This makes process planning for human-robot collaboration more complex and challenging than traditional task scheduling. Currently, researchers primarily use intelligent optimization methods such as reinforcement learning, genetic algorithms, and bee colony algorithms to address dynamic process planning. Although many researchers have proposed effective solutions for multi-objective sequence planning, these approaches are often based on pre-defined data parameters. In actual human-robot collaborative production, when an operator is unable to complete a given process due to fatigue or unexpected circumstances, the system must dynamically adjust the original process to another one, and the operator's movements will also change accordingly. Therefore, monitoring the operator's real-time movements for dynamic planning is essential for human-robot collaborative process planning. Existing fatigue assessment methods fail to fully account for the operator's movement changes in human-robot collaboration, making them difficult to achieve effective results in long, high-load assembly tasks.

[0005] In summary, existing human-robot collaborative assembly sequence planning has two main shortcomings: 1. There is a lack of low-invasive detection methods for the real-time accumulation of operator muscle fatigue, making it difficult to consider the dynamic nature of operator muscle fatigue during assembly sequence planning; 2. There is a lack of comprehensive consideration of the dynamics of assembler operating efficiency and assembler muscle fatigue, resulting in the generated assembly process allocation plan being out of touch with the actual production environment. Summary of the Invention

[0006] In order to solve the above problems, the present invention provides a human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body, comprising the steps of:

[0007] S1: Obtain a set of multi-angle action images of the operator and construct a human skeletal muscle model based on the multi-angle action image set;

[0008] S2: Obtain the operator's biomechanical data set, and construct a muscle fatigue index model using a multi-angle action image set, a human skeletal muscle model, and the biomechanical data set;

[0009] S3: Obtain a set of sample assembly action images, train a deep learning model based on the set of sample assembly action images, and build an action category recognition model;

[0010] S4: Obtain a set of actual assembly action images of the operator in the current human-machine collaborative assembly process. Calculate the fatigue index at each moment and the actual assembly time of the operator using the actual assembly action image set, the human skeletal muscle model, the muscle fatigue index model, and the action category recognition model. Calculate the actual fatigue accumulation value using the fatigue index at each moment.

[0011] S5: Obtain the actual assembly time of the robot, and based on the actual assembly time of the robot, the actual assembly time of the human, and the actual accumulated fatigue value, calculate the optimal population through a multi-objective genetic algorithm, and optimize the human-machine collaborative assembly process through the optimal population.

[0012] Preferably, step S1 is specifically as follows:

[0013] S11: using multiple visual acquisition devices to collect a set of multi-angle action images of the operator from different angles, extracting key skeleton points from the multi-angle action image set to obtain sample key skeleton point data, and performing two-dimensional action posture recognition on the sample key skeleton point data to obtain a set of two-dimensional human action posture data;

[0014] S12: Performing three-dimensional reconstruction on the two-dimensional human body motion posture data set to obtain a three-dimensional human body motion posture data set;

[0015] S13: Perform skeletal muscle simulation modeling based on the human body three-dimensional motion posture data set to obtain a human skeletal muscle model.

[0016] Preferably, step S2 is specifically as follows:

[0017] S21: Acquire a biomechanical data set of the operator, the biomechanical data set including: current muscle load and maximum voluntary muscle contraction value;

[0018] S22: Inputting the multi-angle action image set into the human skeletal muscle model to obtain sample muscle force time series data, and obtaining the current maximum applicable force of the muscle through the sample muscle force time series data;

[0019] S23: Construct a muscle fatigue index model based on the current maximum applicable force of the muscle, the current load of the muscle, and the maximum voluntary contraction value of the muscle.

[0020] Preferred:

[0021] The expression of the muscle fatigue index model is:

[0022]

[0023] Where t represents the current moment, U(t) represents the fatigue index, and F cem (t) represents the current maximum force that the muscle can exert, F load (t) represents the current load of the muscle, and MVC represents the maximum voluntary contraction value of the muscle.

[0024] Preferably, step S3 is specifically as follows:

[0025] S31: Building a deep learning model based on long short-term memory neural network;

[0026] S32: labeling the sample assembly action image set to obtain a sample action category set;

[0027] S33: Iteratively train the deep learning model through the sample assembly action image set and the sample action category set, continuously adjust the parameters of the deep learning model until the loss function converges, and obtain the action category recognition model.

[0028] Preferably, step S4 is specifically as follows:

[0029] S41: Acquire a set of actual assembly action images of the operator in the current human-machine collaborative assembly process through a visual acquisition device, input the set of actual assembly action images into a human skeletal muscle model, and calculate actual muscle force time series data;

[0030] S42: Inputting the actual muscle force time series data into the muscle fatigue index model, calculating the fatigue index at each moment, and adding the fatigue index at each moment to obtain the actual fatigue accumulation value;

[0031] S43: Inputting the actual assembly action image set into the action category recognition model to obtain the actual action category set;

[0032] S44: Obtaining a set of actual joint angle change values through calculation of the actual action category set, and obtaining the actual assembly time of the person through calculation of the actual joint angle change value set.

[0033] Preferably, step S44 is specifically as follows:

[0034] S441: Divide the current human-machine collaborative assembly process into multiple time periods according to preset time intervals;

[0035] S442: Obtaining the actual motion categories at both ends of each time period from the actual motion category set, and calculating the actual joint angle change value between each two adjacent actual motion categories;

[0036] S443: Add up the time intervals of each time period in which the actual joint angle change value is less than the preset value to obtain the actual assembly time of the person.

[0037] Preferably, step S5 is specifically as follows:

[0038] S51: constructing an initial population of the current human-robot collaborative assembly process, and using the initial population as the current population. The current population includes multiple individuals, and the individuals are operators or robots.

[0039] S52: Calculate and obtain the total fatigue index and total time of the current population, where the total fatigue index is the cumulative value of the actual fatigue accumulation values of all operators, and the total time is the cumulative value of the actual assembly time of all operators and the actual assembly time of all robots;

[0040] S53: Calculate the target value by total fatigue index and total time;

[0041] S54: Perform non-dominated sorting on the individuals in the current population according to the target value, and calculate the frontier layer and crowding distance;

[0042] S55: Select, cross and mutate the current population through the frontier layer and crowding distance to obtain a new population;

[0043] S56: If the maximum number of iterations is reached, the new population is used as the optimal population, and the optimized human-machine collaborative assembly process is obtained through the optimal population; otherwise, the new population is used as the current population, and the process returns to step S52.

[0044] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body is implemented.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body is implemented.

[0046] The present invention has the following beneficial effects:

[0047] 1. A muscle fatigue index model personalized for each operator is constructed by using the operator's biomechanical data set, multi-angle motion image set, and human skeletal muscle model. The fatigue index of each operator can be dynamically calculated based on their real-time motion and muscle load. By customizing the muscle fatigue index model for each operator, non-invasive detection of fatigue accumulation values in the assembly process can be achieved.

[0048] 2. The total assembly time is calculated by the actual assembly time of the robot and the actual assembly time of the human. The operator's fatigue accumulation value and the total assembly time of the human and the machine are optimized through a multi-objective genetic algorithm. By non-invasively and in real time monitoring human muscle fatigue and assembly action time, the human-machine collaborative assembly process is dynamically adjusted to improve the efficiency of human-machine collaborative assembly and reduce human muscle fatigue. At the same time, it is more suitable for actual assembly environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0050] Figure 2 This is the structural diagram of the action category recognition model;

[0051] Figure 3 Assign Gantt charts to fix muscle fatigue and assembly time-consuming processes;

[0052] Figure 4 Gantt chart for real-time updating of muscle fatigue and assembly time-consuming process of the present invention;

[0053] Figure 5 A total time comparison diagram of the planning algorithm of the present invention that considers real-time muscle fatigue and assembly time and the existing planning algorithm that fixes muscle fatigue and assembly time;

[0054] Figure 6 A comparison diagram of the accumulated total fatigue between the planning algorithm of the present invention that considers real-time muscle fatigue and assembly time consumption and the existing planning algorithm that fixes muscle fatigue and assembly time consumption;

[0055] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0056] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0057] The following will be combined with the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] Reference Figure 1 The present invention provides a human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body, comprising the steps of:

[0059] S1: Obtain a set of multi-angle action images of the operator and construct a human skeletal muscle model based on the multi-angle action image set;

[0060] In some embodiments:

[0061] Step S1 is specifically as follows:

[0062] S11: using multiple visual acquisition devices to collect a set of multi-angle action images of the operator from different angles, extracting key skeleton points from the multi-angle action image set to obtain sample key skeleton point data, and performing two-dimensional action posture recognition on the sample key skeleton point data to obtain a set of two-dimensional human action posture data;

[0063] In some embodiments, RTMPose is used to capture motion images of a person from different angles using three cameras. Based on the captured motion images of the person, the Halpe26 model of RTMPose is used to extract full-body image information. 26 sets of key bone point data are obtained for each image, and information on points 5 (left shoulder), 6 (right shoulder), 7 (right elbow), 9 (right wrist), and 18 (neck) in the Halpe26 model are extracted.

[0064] The two-dimensional coordinate system parameters of each camera are as follows:

[0065]

[0066] Then calibrate the internal and external parameters of each camera to obtain the internal parameter matrix K i , rotation matrix R i and the translation vector t i , based on these parameters, the camera projection matrix P can be obtained i :

[0067]

[0068] S12: Performing three-dimensional reconstruction on the two-dimensional human body motion posture data set to obtain a three-dimensional human body motion posture data set;

[0069] In some embodiments, the multi-view visual acquisition and 3D reconstruction technology of the Pose2Sim library is used to synthesize 2D motion pose data into 3D motion pose data;

[0070] Use Pose2Sim to synthesize the poses extracted by the three cameras into high-precision 3D pose data X, Y, Z, 1):

[0071]

[0072] S13: Perform skeletal muscle simulation modeling based on the human body three-dimensional motion posture data set to obtain a human skeletal muscle model.

[0073] In some embodiments, a personalized human skeletal muscle model is constructed based on the highly simulated body and right arm skeletal muscle models using the zoom function of the OpenSim software and the obtained high-precision three-dimensional human motion data.

[0074] S2: Obtain the operator's biomechanical data set, and construct a muscle fatigue index model using a multi-angle action image set, a human skeletal muscle model, and the biomechanical data set;

[0075] In some embodiments:

[0076] Step S2 is specifically as follows:

[0077] S21: Acquire a biomechanical data set of the operator, the biomechanical data set including: current muscle load and maximum voluntary muscle contraction value;

[0078] In some embodiments, the current load F of the muscle load (t) and the maximum voluntary muscle contraction value (MVC) are substituted according to the biomechanical data of the test operator. The maximum voluntary muscle contraction value (MVC) of a person is calculated as follows: the test subject bears the load in a way that can best activate the affected muscles in the arm, and the load is continuously increased until it can no longer be tolerated. This experiment is repeated 5 times, and the average value is taken as the maximum voluntary muscle contraction value. load The calculation formula of (t) is as follows:

[0079] F load =F external +F arm

[0080] Among them, F external F is the weight of the parts to be grabbed during assembly. arm is the weight of the operator's arm;

[0081] S22: Inputting the multi-angle action image set into the human skeletal muscle model to obtain sample muscle force time series data, and obtaining the current maximum applicable force of the muscle through the sample muscle force time series data;

[0082] S23: Construct a muscle fatigue index model based on the current maximum applicable force of the muscle, the current load of the muscle, and the maximum voluntary contraction value of the muscle.

[0083] In some embodiments, the formula of the muscle fatigue index model describes the rate of change of the fatigue index over time. The increase in the fatigue index depends on the current load of the muscle, the maximum voluntary contraction (MVC) of the muscle, and the current maximum force that the muscle can exert;

[0084] The expression of the muscle fatigue index model is:

[0085]

[0086] Where t represents the current moment, U(t) represents the fatigue index, and F cem (t) represents the current maximum force that the muscle can exert, F load (t) represents the current load of the muscle, and MVC represents the maximum voluntary contraction value of the muscle.

[0087] In some embodiments, muscle fatigue is closely related to a decrease in the maximum force that can be output. The current maximum force that can be applied by a muscle decreases as the current load on the muscle increases. The rate of change of the current maximum force that can be applied by a muscle over time can be expressed as follows:

[0088]

[0089] S3: Obtain a set of sample assembly action images, train a deep learning model based on the set of sample assembly action images, and build an action category recognition model;

[0090] In some embodiments:

[0091] Step S3 is specifically as follows:

[0092] S31: Building a deep learning model based on long short-term memory neural network;

[0093] S32: labeling the sample assembly action image set to obtain a sample action category set;

[0094] In some embodiments, taking the scenario of human-machine collaborative assembly of a reduction gearbox as an example, a total of 10 labels are given to the actions in this scenario, namely upper left a, upper right a, lower left a, lower right a, middle a, upper left b, upper right b, lower left b, lower right b, middle b, among which upper left a refers to a person grabbing part a in the upper left corner, and so on.

[0095] S33: Iteratively train the deep learning model through the sample assembly action image set and the sample action category set, continuously adjust the parameters of the deep learning model until the loss function converges, and obtain the action category recognition model.

[0096] In some embodiments, a training data set is constructed with a time step of 0.2s. The data is normalized (θ1 and θ2 are the angles of the shoulder joint and elbow joint), where μ and σ are the mean and standard deviation of each feature in all samples:

[0097]

[0098] The LSTM layer of the deep learning model is calculated through multiple iterations and based on the memory unit C t AND output gate o t Output the hidden state h at each time step t , the calculation formula is as follows:

[0099]

[0100] Among them, σ is the Sigmoid activation function, O t is the output of the output gate, bo is the bias of the output gate, W o is the weight matrix of the output gate.

[0101] Based on the constructed data set, the long short-term memory neural network is trained to classify human movements according to the human joint angles. Then, based on the recognized movements and the muscle fatigue index of each type of movement, the muscle fatigue index of the movement is obtained in real time. After the training is completed, the structure of the movement category recognition model is as follows: Figure 2 shown.

[0102] The loss used in training is BCE With Logits Loss (Binary Cross Entropy with Logits), which is calculated as follows:

[0103]

[0104] Among them, y i is the actual label, is the predicted probability value. By minimizing the loss function, the model will continuously adjust its weight parameters so that the gap between the predicted results and the true labels gradually decreases, thereby improving the accuracy of the model.

[0105] The present invention selects Adam optimizer to train the data set in order to improve the training efficiency and performance of the model. Its algorithm is as follows:

[0106]

[0107] S4: Obtain a set of actual assembly action images of the operator in the current human-machine collaborative assembly process. Calculate the fatigue index at each moment and the actual assembly time of the operator using the actual assembly action image set, the human skeletal muscle model, the muscle fatigue index model, and the action category recognition model. Calculate the actual fatigue accumulation value using the fatigue index at each moment.

[0108] In some embodiments,

[0109] Step S4 is specifically as follows:

[0110] S41: Acquire a set of actual assembly action images of the operator in the current human-machine collaborative assembly process through a visual acquisition device, input the set of actual assembly action images into a human skeletal muscle model, and calculate actual muscle force time series data;

[0111] In some embodiments, the operator performs the assembly steps assigned to the operator in the algorithm, while the RGB camera is used to capture these actions and extract relevant data. For example, in the scenario of human-robot collaborative assembly of a reduction gearbox, the operator and the collaborative robot need to work together to assemble each part to the main shaft of the reduction gearbox in sequence. In order to enable the collaborative robot to respond immediately to changes in the process sequence after the process is freely assigned, the corresponding robot trajectory has been pre-planned for each process that can be completed by the robot;

[0112] The human skeletal muscle model OpenSim calculates muscle force changes according to the following formula:

[0113] F=F max ×f L (L)×f V (v)

[0114] Among them, F max represents the maximum isometric contraction force of the muscle, f L (L) represents the length-force relationship function, f V (v) represents the velocity-force relationship function;

[0115] S42: Inputting the actual muscle force time series data into the muscle fatigue index model, calculating the fatigue index at each moment, and adding the fatigue index at each moment to obtain the actual fatigue accumulation value;

[0116] S43: Inputting the actual assembly action image set into the action category recognition model to obtain the actual action category set;

[0117] In some embodiments, a pre-trained deep learning model, trained on real-world data from operators performing assembly tasks, identifies the action type based on changes in joint angles. The model identifies the action corresponding to the recorded joint angle changes and updates the fatigue index and duration of each action in real time. The fatigue index can then be used to calculate the cumulative fatigue value.

[0118] S44: Obtaining a set of actual joint angle change values through calculation of the actual action category set, and obtaining the actual assembly time of the person through calculation of the actual joint angle change value set.

[0119] In some embodiments, when the operator performs the assembly process, the camera will be activated and record their movements. After the recording is completed, OpenPose is used to extract the key skeletal points in the movement, and the angles of the shoulder and elbow joints are calculated based on these key points;

[0120] Step S44 is specifically as follows:

[0121] S441: Divide the current human-machine collaborative assembly process into multiple time periods according to preset time intervals;

[0122] S442: Obtaining the actual motion categories at both ends of each time period from the actual motion category set, and calculating the actual joint angle change value between each two adjacent actual motion categories;

[0123] In some embodiments, taking the calculation of the change value of the shoulder joint as an example, the key bone point data is imported into OpenSim to calculate the joint angle change, wherein the neck coordinates are marked as (x1, y1), the right shoulder is marked as (x2, y2), and the right elbow is marked as (x3, y3). The calculation formula of the shoulder joint θ is as follows:

[0124]

[0125] S443: Add up the time intervals of each time period in which the actual joint angle change value is less than the preset value to obtain the actual assembly time of the person.

[0126] In some embodiments, the duration of the action is determined by the change in the elbow joint angle over time. When the elbow joint is monitored to change by less than 2 degrees within 2 seconds, the action is considered completed, the time is recorded, and the duration is updated.

[0127] S5: Obtain the actual assembly time of the robot, and based on the actual assembly time of the robot, the actual assembly time of the human, and the actual accumulated fatigue value, calculate the optimal population through a multi-objective genetic algorithm, and optimize the human-machine collaborative assembly process through the optimal population.

[0128] In some embodiments, when a complete assembly task is completed, the system will replace the default values with new fatigue and time data, and use the updated data to re-optimize the planning of the assembly sequence through a multi-objective genetic algorithm (Non-dominated Sorting Genetic Algorithm II, NSGA-II). The present invention applies a multi-objective optimization algorithm to process scheduling problems, aiming to achieve dual optimization of time and muscle fatigue. Through the NSGA-II algorithm, the allocation scheme of different tasks (assigned to "people" or "robots") is calculated to optimize multiple objectives at the same time. Ultimately, the algorithm outputs the Pareto front of the optimal solution and the crowding distance of each solution.

[0129] Step S5 is specifically as follows:

[0130] S51: constructing an initial population of the current human-robot collaborative assembly process, and using the initial population as the current population. The current population includes multiple individuals, and the individuals are operators or robots.

[0131] In some embodiments, the "population initialization" process is first performed to generate an initial population of 50 individuals. Each individual is a sequence of "human" or "robot," representing a task assignment scheme. For tasks of the "human / robot" category, "human" or "robot" is randomly selected; for tasks that can only be completed by humans, "human" is fixed. The formula is as follows:

[0132] Individual i =random.choice("human","robot")

[0133] Among them, Individual i represents the task allocation plan for the i-th individual.

[0134] S52: Calculate and obtain the total fatigue index and total time of the current population, where the total fatigue index is the cumulative value of the actual fatigue accumulation values of all operators, and the total time is the cumulative value of the actual assembly time of all operators and the actual assembly time of all robots;

[0135] In some embodiments, for each task, if the task is performed by a robot, its execution time is added to the total time; if it is performed by a person, the person's execution time is added to the total time. The fatigue index is calculated only when a person performs the task, and the fatigue index remains unchanged when the robot performs the task (that is, the increase is 0). In this way, the algorithm can comprehensively consider the two goals of time and fatigue, and provide basic data support for the subsequent optimization process. The specific calculation process is as follows, where 1 robot Indicates that the task is performed by the robot, 1 human Indicates that the task is performed by a person:

[0136]

[0137] S53: Calculate the target value by total fatigue index and total time;

[0138] In some embodiments, the target value is the sum of the total fatigue index (total fatigue) and the total time (total time);

[0139] S54: Perform non-dominated sorting on the individuals in the current population according to the target value, and calculate the frontier layer and crowding distance;

[0140] In some embodiments, the algorithm performs non-dominated sorting on the individuals in the population according to the calculated target values to determine different frontier layers (Pareto Front). The core idea of non-dominated sorting is to judge the dominance relationship between individuals: if an individual is not inferior to another individual in all objectives, and is strictly superior to another individual in at least one objective, the former is considered to dominate the latter. Based on this rule, the frontier solution consists of individuals that are not dominated by any other individual, thus forming a set of non-dominated solution sets. Specifically, the necessary and sufficient condition for individual i to dominate individual j is that individual i is not inferior to individual j in all objective values, and is strictly superior to individual j in at least one objective value. The judgment of this dominance relationship provides the basis for subsequent multi-objective optimization, ensuring that the algorithm can effectively screen out potential solution sets. The calculation formula of the frontier layer is as follows:

[0141] and

[0142] To maintain solution diversity and prevent excessive concentration of solutions on the Pareto front, the algorithm further calculates the crowding distance of each individual in its frontier layer. The crowding distance is used to measure the distribution density of solutions within its neighborhood. The larger the distance, the sparser the other solutions around the solution, which helps maintain solution diversity. The calculation formula of the crowding distance is as follows:

[0143]

[0144] Among them, d i represents the crowding distance of individual i, Obj k Indicates the value of the kth target, max k and min k They represent the maximum and minimum values of the target among all solutions respectively.

[0145] S55: Select, cross and mutate the current population through the frontier layer and crowding distance to obtain a new population;

[0146] In some embodiments, the algorithm then enters the genetic operation phase, which includes three key processes: selection, crossover, and mutation. First, the selection process is performed to select parent individuals from the current population for the subsequent crossover operation.

[0147] After selecting the parent individuals, two parent individuals are randomly selected under a certain crossover probability, and two new offspring individuals are generated through crossover operation. The calculation formula is as follows:

[0148] Child1, Child2=Crossover(parent1, parent2)

[0149] Finally, the algorithm performs a mutation process to further increase the diversity of the population and avoid falling into local optimality. The mutation operation is performed according to the following formula:

[0150] mutate(Individual i )=flip(gene i )

[0151] if random <mutaterate

[0152] S56: If the maximum number of iterations is reached, the new population is used as the optimal population, and the optimized human-machine collaborative assembly process is obtained through the optimal population; otherwise, the new population is used as the current population, and the process returns to step S52.

[0153] In some embodiments, the execution process of the multi-objective genetic algorithm of the present invention is as follows: 1. Initialize the population; 2. Iterate the selection, crossover, and mutation operations to continuously optimize the objectives; 3. At the end of each generation, calculate the target values and crowding distances of all individuals, perform non-dominated sorting, and update the population. The optimal population outputted at the end is the Pareto front solution, which contains a set of optimal equilibrium solutions for the total fatigue index and the total time consumption. The core idea of the present invention using this algorithm is to find the Pareto front solution by simulating the evolutionary process, continuously optimizing the two objectives of the total fatigue index and the total time consumption, and ensuring the diversity of the solution through the crowding distance.

[0154] After each round of assembly tasks, the updated fatigue index and time consumption data are input into the NSGA-II algorithm to replan the assembly sequence for the next round. To verify the superiority of the present invention, the real-time human movements of five test operators were monitored, and the fatigue index and time consumption of the assembly actions were recorded in real time. The cumulative total fatigue and total time consumption of the multi-objective genetic algorithm of the present invention were compared with those of the existing NSGA-II algorithm. The parameters of the assembly experiment are shown in Table 1.

[0155] Table 1 Parameters of the assembly test case

[0156]

[0157] The existing fixed muscle fatigue and assembly time-consuming process allocation Gantt chart is as follows Figure 3 As shown, the real-time updated muscle fatigue and assembly time-consuming process allocation Gantt chart of the present invention is as follows Figure 4 As shown in the figure, the total time comparison of the planning algorithm considering real-time muscle fatigue and assembly time of the present invention and the planning algorithm with fixed muscle fatigue and assembly time is shown in the figure Figure 5 As shown in the figure, the cumulative total fatigue comparison of the planning algorithm of the present invention considering real-time muscle fatigue and assembly time consumption and the existing planning algorithm of fixed muscle fatigue and assembly time consumption is shown in the figure. Figure 6As shown in the figure, the assembly time of the five test operators exhibited similar trends. From rounds 1 to 4, the assembly time of both algorithms gradually decreased. From rounds 5 to 9, the time of the real-time update algorithm increased significantly in the fifth round after the weights were updated, and then gradually decreased in subsequent rounds. The assembly time of the fixed parameter algorithm increased with each round. Finally, in the tenth round, the time consumed by operators A, C, and D under the real-time update algorithm was less than that of the fixed parameter algorithm, while the time consumed by operators B and E under the real-time update algorithm was also close to that of the fixed parameter algorithm. Regarding the fatigue index accumulation of the assembly task, due to the real-time monitoring and updating of the operator's fatigue index in the algorithm, the fatigue index accumulation of the five operators under the real-time update algorithm was less than that of the fixed parameter algorithm at each assembly round. From rounds 1 to 4, the fatigue index accumulation of both algorithms was relatively small. From rounds 5 to 9, the fatigue accumulation gap between the real-time update algorithm with adjusted weights and the fixed parameter algorithm widened further. Finally, in the tenth round, the fatigue accumulation of the five test operators under the real-time update algorithm was less than that under the fixed parameter algorithm.

[0158] In summary, during long-term assembly work, the multi-objective genetic algorithm proposed in the present invention monitors human fatigue and process time in real time and updates the algorithm. The assembly efficiency of the real-time updated algorithm is better than that of the traditional fixed-parameter algorithm, and it can effectively reduce operator fatigue.

[0159] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 50 provided in an embodiment of the present application includes a memory 51 and a processor 52. The memory 51 stores a computer program, wherein the computer program, when executed by the processor, implements the human-machine collaborative assembly planning method based on real-time human muscle fatigue.

[0160] Specifically, the processor 52 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 52 may also include onboard memory for caching purposes. The processor 52 may be a single processing unit or multiple processing units for executing different actions of the method flow according to the embodiments of the present application.

[0161] Memory 51 can be, for example, any medium capable of containing, storing, conveying, disseminating, or transmitting instructions. For example, memory 51 can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, components, or communication media. Specific examples of memory 51 include: magnetic storage devices, such as magnetic tape or hard disk drives (HDDs); optical storage devices, such as compact discs (CD-ROMs); random access memory (RAM) or flash memory; and / or wired or wireless communication links.

[0162] The present application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the human-machine collaborative assembly planning method based on real-time human muscle fatigue. The computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not be incorporated into the device / apparatus / system. The computer-readable medium carries one or more programs, and when executed, implements the method of the embodiments of the present application.

[0163] According to an embodiment of the present application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, radio frequency signals, or any suitable combination thereof.

[0164] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above-mentioned embodiments, but should be determined not only by the attached claims, but also by the equivalents of the attached claims. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body, characterized in that: Including steps: S1: Obtain a set of multi-angle action images of the operator and construct a human skeletal muscle model based on the multi-angle action image set; S2: Obtain the operator's biomechanical data set, and construct a muscle fatigue index model using a multi-angle action image set, a human skeletal muscle model, and the biomechanical data set; S3: Obtain a set of sample assembly action images, train a deep learning model based on the set of sample assembly action images, and build an action category recognition model; S4: Obtain a set of actual assembly action images of the operator in the current human-machine collaborative assembly process. Calculate the fatigue index at each moment and the actual assembly time of the operator using the actual assembly action image set, the human skeletal muscle model, the muscle fatigue index model, and the action category recognition model. Calculate the actual fatigue accumulation value using the fatigue index at each moment. S5: Obtain the actual assembly time of the robot, and based on the actual assembly time of the robot, the actual assembly time of the human, and the actual accumulated fatigue value, calculate the optimal population through a multi-objective genetic algorithm, and optimize the human-machine collaborative assembly process through the optimal population.

2. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 1 is characterized in that: Step S1 is specifically as follows: S11: using multiple visual acquisition devices to collect a set of multi-angle action images of the operator from different angles, extracting key skeleton points from the multi-angle action image set to obtain sample key skeleton point data, and performing two-dimensional action posture recognition on the sample key skeleton point data to obtain a set of two-dimensional human action posture data; S12: Performing three-dimensional reconstruction on the two-dimensional human body motion posture data set to obtain a three-dimensional human body motion posture data set; S13: Perform skeletal muscle simulation modeling based on the human body three-dimensional motion posture data set to obtain a human skeletal muscle model.

3. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 1 is characterized in that: Step S2 is specifically as follows: S21: Acquire a biomechanical data set of the operator, the biomechanical data set including: current muscle load and maximum voluntary muscle contraction value; S22: Inputting the multi-angle action image set into the human skeletal muscle model to obtain sample muscle force time series data, and obtaining the current maximum applicable force of the muscle through the sample muscle force time series data; S23: Construct a muscle fatigue index model based on the current maximum applicable force of the muscle, the current load of the muscle, and the maximum voluntary contraction value of the muscle.

4. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 3 is characterized by: The expression of the muscle fatigue index model is: Where t represents the current moment, U(t) represents the fatigue index, and F cem (t) represents the current maximum force that the muscle can exert, F load (t) represents the current load of the muscle, and MVC represents the maximum voluntary contraction value of the muscle.

5. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 1 is characterized in that: Step S3 is specifically as follows: S31: Building a deep learning model based on long short-term memory neural network; S32: labeling the sample assembly action image set to obtain a sample action category set; S33: Iteratively train the deep learning model through the sample assembly action image set and the sample action category set, continuously adjust the parameters of the deep learning model until the loss function converges, and obtain the action category recognition model.

6. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 1 is characterized in that: Step S4 is specifically as follows: S41: Acquire a set of actual assembly action images of the operator in the current human-machine collaborative assembly process through a visual acquisition device, input the set of actual assembly action images into a human skeletal muscle model, and calculate actual muscle force time series data; S42: Inputting the actual muscle force time series data into the muscle fatigue index model, calculating the fatigue index at each moment, and adding the fatigue index at each moment to obtain the actual fatigue accumulation value; S43: Inputting the actual assembly action image set into the action category recognition model to obtain the actual action category set; S44: Obtaining a set of actual joint angle change values through calculation based on the actual action category set, and obtaining the actual assembly time of the person through calculation based on the actual joint angle change value set.

7. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 6 is characterized in that: Step S44 is specifically as follows: S441: Divide the current human-machine collaborative assembly process into multiple time periods according to preset time intervals; S442: Obtaining the actual motion categories at both ends of each time period from the actual motion category set, and calculating the actual joint angle change value between each two adjacent actual motion categories; S443: Add up the time intervals of each time period in which the actual joint angle change value is less than the preset value to obtain the actual assembly time of the person.

8. The human-machine collaborative assembly planning method based on real-time human muscle fatigue according to claim 1 is characterized in that: Step S5 is specifically as follows: S51: constructing an initial population of the current human-robot collaborative assembly process, and using the initial population as the current population. The current population includes multiple individuals, and the individuals are operators or robots. S52: Calculate and obtain the total fatigue index and total time of the current population, where the total fatigue index is the cumulative value of the actual fatigue accumulation values of all operators, and the total time is the cumulative value of the actual assembly time of all operators and the actual assembly time of all robots; S53: Calculate the target value by total fatigue index and total time; S54: Perform non-dominated sorting on the individuals in the current population according to the target value, and calculate the frontier layer and crowding distance; S55: Select, cross and mutate the current population through the frontier layer and crowding distance to obtain a new population; S56: If the maximum number of iterations is reached, the new population is used as the optimal population, and the optimized human-machine collaborative assembly process is obtained through the optimal population; otherwise, the new population is used as the current population, and the process returns to step S52.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body as described in any one of claims 1 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the human-machine collaborative assembly planning method based on real-time muscle fatigue of the human body as claimed in any one of claims 1 to 8 is implemented.

Images