Man-machine integrated simulation deduction method and system for mechanical arm control

Abstract

The invention provides a man-machine integrated simulation deduction method and system oriented to mechanical arm control, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining the current pose and target pose of a target person of a man-machine tight coupling system, the current state of a mechanical arm, and the target pose of the target person; the current minimum distance between the whole mechanical arm and the target person and the target object is obtained; the obtained data are input into a decision model, and a control instruction, output by the decision model, of the mechanical arm is obtained; performing simulation deduction on the man-machine tight coupling system to obtain a simulation result, and optimizing the man-machine tight coupling system based on the simulation result; wherein the decision model is obtained by training based on sample data; the sample data is obtained by amplifying original sample data based on a generative artificial intelligence technology. According to the method, the control instruction is predicted based on the decision model, the man-machine tight coupling system is optimized through simulation deduction, the flexibility is high, and the method is suitable for complex application scenes.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a human-machine integrated simulation deduction method and system for robotic arm manipulation. Background Art

[0002] Against the backdrop of growing scientific and technological strength, research in specialized fields such as aerospace, deep earth and deep sea exploration, and nuclear power has become a crucial initiative in promoting China's development into a technological powerhouse. These key areas face extremely complex environments and demanding technical challenges, posing significant safety risks. Making accurate decisions in complex scenarios is a critical issue that needs to be addressed. Currently, existing simulation methods use only a small number of unrepresentative samples to train decision-making models, resulting in poor flexibility and adaptability, making them difficult to adapt to complex application scenarios. Summary of the Invention

[0003] The present invention provides a human-machine integrated simulation and deduction method and system for robotic arm manipulation, which is used to solve the defects of the simulation and deduction method in the prior art, which only uses a small number of unrepresentative sample training models, has poor flexibility and adaptability, and is difficult to adapt to complex application scenarios.

[0004] The present invention provides a human-machine integration simulation deduction method for manipulating a robotic arm, comprising: Obtaining the current and target poses of the target person in the human-machine tightly coupled system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object; The current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object are input into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model; the control instruction is used to control the robotic arm to transfer the target person to the target position and avoid collision between the robotic arm and the target object during the transfer process; Performing simulation on the human-machine tightly coupled system to obtain simulation results, and optimizing the human-machine tightly coupled system based on the simulation results; Among them, the decision model is obtained by training based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: the sample posture and sample target posture of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

[0005] In some embodiments, inputting the current posture and target posture of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model includes: Inputting the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object into a pre-built decision model; Based on the decision model, the motion path of the robotic arm is predicted according to the current posture and target posture of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object, and a collision-free predicted path of the robotic arm is obtained. Based on the collision-free predicted path, the control instructions of the robotic arm are determined and the control instructions of the robotic arm are output.

[0006] In some embodiments, predicting the motion path of the robotic arm to obtain a collision-free predicted path of the robotic arm includes: The motion path of the robotic arm is predicted based on the constraint that the predicted minimum distance between the robotic arm, the target person, and the target object is greater than a preset distance threshold, thereby obtaining a collision-free predicted path for the robotic arm. The predicted value of the minimum distance is estimated based on the predicted state of the robotic arm, the predicted posture of the target person, and the predicted posture of the target object.

[0007] In some embodiments, the current state of the robotic arm includes: the current angles of the joints of the robotic arm, the current pitch angle and the current yaw angle of the gimbal where the robotic arm is located.

[0008] In some embodiments, optimizing the human-machine tightly coupled system based on the simulation results includes: Based on the simulation results, identifying abnormal conditions and determining key variables related to the abnormal conditions; Performing a risk assessment on the human-machine tightly coupled system based on the key variables to obtain a risk assessment result; Based on the risk assessment result, the human-machine tightly coupled system is optimized to obtain an optimized human-machine tightly coupled system; The abnormal situation includes at least one of the following: The target person fails to reach the target posture; The robotic arm collides with the target object during the process of transferring the target person; The target person collides with the target object while being transferred by the robotic arm.

[0009] In some embodiments, the training process of the decision model includes: Constructing an initial decision model, determining a variable set and a rule set of the initial decision model, wherein the rule set includes a plurality of original inference rules corresponding to the variable set; Acquire original sample data, the original sample data including: an original sample pose and an original sample target pose of an original sample target person, an original sample state of an original sample manipulator, and an original sample minimum distance between the original sample manipulator and the original sample target person as a whole and an original sample target object; Amplifying the original sample data based on generative artificial intelligence technology to obtain the sample data; The sample data is used as a training sample to train the initial decision model, and after the training is completed, the decision model is obtained.

[0010] In some embodiments, constructing an initial decision model and determining a variable set and a rule set of the initial decision model include: Acquire experimental data from human subjects in specific task scenarios, including: human operation data, eye movement data, physiological signals reflecting human status, machine state change data, operation videos, and interview data on cognitive decision-making thinking; Based on the experimental data, a cognitive architecture is used to formally express human cognitive decision-making behavior, an initial decision model is constructed, and a variable set and a rule set of the initial decision model are determined.

[0011] In some embodiments, amplifying the original sample data based on generative artificial intelligence technology to obtain the sample data includes: Inputting the original sample data and noise into a pre-built Generative Artificial Intelligence (GAI) model to obtain new sample data output by the GAI model; Aggregating the original sample data and the new sample data to obtain the sample data; The training process of the GAI model includes: Acquire training sample data and training noise, and determine a training sample data label corresponding to the training sample data, wherein the training sample data corresponds to the original sample data; Inputting the training sample data and training noise into a pre-built initial GAI model to obtain a prediction result of the training sample data output by the initial GAI model; The training sample data labels are compared with the prediction results of the training sample data, and the parameters of the initial GAI model are iteratively optimized to obtain the GAI model.

[0012] In some embodiments, the training of the initial decision model includes: Inputting the sample pose and sample target pose of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm, the sample target person, and the sample target object into the initial decision model, and obtaining a predictive control instruction for the robotic arm output by the initial decision model; After the robotic arm executes the predictive control instruction, calculating a reward value of the predictive control instruction; Iteratively optimize the parameters of the initial decision model according to the reward value of the predictive control instruction.

[0013] The present invention also provides a human-machine integrated simulation and deduction system for manipulating a robotic arm, comprising: an acquisition unit, configured to acquire the current posture and target posture of the target person in the human-machine tight coupling system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object; a decision unit, configured to input the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model, and obtain a control instruction for the robotic arm output by the decision model; the control instruction is used to control the robotic arm to transfer the target person to the target position and to prevent the robotic arm, the target person, and the target object from colliding during the transfer process; a simulation deduction unit, configured to simulate and deduce the human-machine tightly coupled system to obtain simulation results, and optimize the human-machine tightly coupled system based on the simulation results; Among them, the decision model is obtained by training based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: the sample posture and sample target posture of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, a human-machine integrated simulation and deduction method for robotic arm manipulation as described in any one of the above-mentioned methods is implemented.

[0015] The human-machine integration simulation and deduction method and system for manipulating a robotic arm provided by the present invention obtains the current and target positions of the target person in the human-machine tightly coupled system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object; inputs the obtained data into a decision model to obtain control instructions for the robotic arm output by the decision model; simulates and deduces the human-machine tightly coupled system to obtain simulation results, and optimizes the human-machine tightly coupled system based on the simulation results; wherein the decision model is obtained by training based on sample data; and the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology. The present invention predicts control instructions based on the decision model and optimizes the human-machine tightly coupled system through simulation and deduction. It has strong flexibility and adaptability and is suitable for complex application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is one of the flow charts of the human-machine integration simulation and deduction method for robotic arm manipulation provided by an embodiment of the present invention.

[0018] Figure 2 This is the second flow chart of the human-machine integration simulation and deduction method for robotic arm control provided by an embodiment of the present invention.

[0019] Figure 3 It is a flowchart of the training process of the decision model provided by an embodiment of the present invention.

[0020] Figure 4 It is a flowchart of the modeling and simulation deduction process of the human-machine tightly coupled system provided by an embodiment of the present invention.

[0021] Figure 5 It is a structural diagram of a human-machine integrated simulation and deduction system for robotic arm manipulation provided by an embodiment of the present invention.

[0022] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0024] The terms "first," "second," and the like, as used herein, are used to distinguish similar objects, and are not intended to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable, where appropriate, so that embodiments of the present invention can be implemented in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first" and "second" generally refer to a class of objects and do not limit the number of objects. For example, the first object can be one or more. Furthermore, the term "and / or" as used herein refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0025] Figure 1 This is a flow chart of a human-machine integration simulation method for manipulating a robotic arm provided by an embodiment of the present invention. Figure 1 As shown, a human-machine integration simulation method for manipulating a robotic arm is provided, comprising the following steps: step 110, step 120, and step 130. The steps of the method flow are merely a possible implementation of the present invention.

[0026] Step 110: Obtain the current posture and target posture of the target person in the human-machine tight coupling system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object.

[0027] Optionally, a difference between the current posture of the target person and the target posture is calculated based on the current posture of the target person and the target posture.

[0028] The difference between the current posture and the target posture includes: the difference between the current position and the target position, and the difference between the current posture angle and the target posture angle.

[0029] The current minimum distance refers to the straight-line distance between the current nearest point of the boundary of the robotic arm and the target person as a whole and the target object.

[0030] Optionally, the surfaces of the robotic arm (including the target person) and the target object are decomposed into triangular meshes, two hierarchical envelopes are constructed, and the spatial straight-line distance between the closest triangular meshes on the two surfaces is calculated in real time to obtain the current minimum distance.

[0031] Among them, a tightly coupled human-machine system refers to a system in which humans and machines are highly dependent on each other, influence each other, and work closely together to complete tasks; for example, a tightly coupled human-machine system includes a target person, a robotic arm, and a target object.

[0032] The target object may be a target obstacle in the rescue environment, and the target object may be static or dynamic.

[0033] For example, in a space station extravehicular rescue scenario, astronaut 1 (the target person) is located at the end of the robotic arm, and the rescue mission is: control the movement of the robotic arm within the specified time, transport astronaut 1 to the target position to rescue astronaut 2, and no collision with the space station cabin (the target object) occurs during this period.

[0034] It should be noted that this decision-making model can be applied to tightly coupled human-machine systems in special fields such as aerospace, deep earth and deep sea, and nuclear power, and is suitable for a variety of complex application scenarios.

[0035] In some embodiments, the current state of the robotic arm includes: the current angles of the joints of the robotic arm, the current pitch angle and the current yaw angle of the gimbal where the robotic arm is located.

[0036] It should be noted that precise control of the angles of each joint of the robotic arm is a prerequisite for achieving complex movements and precise operations; the adjustment of the pitch angle enables the robotic arm to change its posture and working range in the vertical direction; the adjustment of the yaw angle is of great significance for the positioning and steering of the robotic arm in the horizontal direction.

[0037] Step 120: Input the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model; the control instruction is used to control the robotic arm to transfer the target person to the target position and avoid collision between the robotic arm, the target person, and the target object during the transfer process; Among them, the decision model is trained based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: the sample posture and sample target posture of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

[0038] The control instructions include at least parameters such as the motion angle, speed, acceleration, etc. of the target joint of the robotic arm.

[0039] Optionally, a large amount of sample data can be collected in simulated or actual rescue scenarios and amplified. Through the diverse sample data, the decision model can learn the control laws in different scenarios.

[0040] Step 130: simulate and deduce the human-machine tightly coupled system to obtain simulation results, and optimize the human-machine tightly coupled system based on the simulation results.

[0041] It's important to note that the decision-making model is trained using a large amount of sample data, making it highly flexible and adaptable. By learning from this sample data, the decision-making model can establish a mapping relationship between input information and robotic arm control commands. This allows it to quickly and accurately output corresponding robotic arm control commands based on the input information in actual rescue scenarios.

[0042] It can be understood that the output control instructions are used to control the robotic arm to transfer the target person to the target position; during the transfer process, the decision model dynamically adjusts the control instructions based on the real-time input target person's posture difference, the robotic arm status and the target object position to ensure that the robotic arm's movement path avoids the target object and avoids collision, while ensuring the smooth and safe transfer of the target person, thereby improving the safety of the rescue.

[0043] In an embodiment of the present invention, the current position and target position of the target person of the human-machine tightly coupled system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object are obtained; the obtained data is input into a decision model to obtain control instructions for the robotic arm output by the decision model; the human-machine tightly coupled system is simulated and deduced to obtain simulation results, and the human-machine tightly coupled system is optimized based on the simulation results; wherein the decision model is obtained by training based on sample data; and the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology. The present invention predicts control instructions based on the decision model and optimizes the human-machine tightly coupled system through simulation and deduction. It has strong flexibility and adaptability, is suitable for complex application scenarios, and improves the safety of the human-machine tightly coupled system.

[0044] Figure 2 The second flow chart of the human-machine integration simulation deduction method for manipulating a robotic arm provided by an embodiment of the present invention. Figure 2 As shown, a human-machine integration simulation method for manipulating a robotic arm is provided, comprising the following steps: Obtain the current state of the robotic arm, determine the current and target poses of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object; Determine whether the control instructions of the robotic arm can be fine-tuned. If so, determine whether the target person has reached the target posture. Otherwise, make rough adjustments to the control instructions of the robotic arm. When it is determined that the target person has not reached the target posture, the control instructions of the robotic arm are fine-tuned.

[0045] Optionally, the current posture and target posture of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object are input into a pre-built decision model to obtain the control instructions of the robotic arm output by the decision model.

[0046] Optionally, in determining whether the control instructions of the robotic arm can be fine-tuned, the joint angle constraints, speed and acceleration constraints, load capacity constraints of the robotic arm can be considered, and time constraints and safety constraints can also be considered to avoid causing harm to the target person.

[0047] Optionally, it is determined whether the angles of the joints of the robotic arm after fine-tuning exceed their physical limit angles; if fine-tuning will cause any joint to exceed its maximum allowable angle or be less than its minimum allowable angle, then such fine-tuning is not feasible.

[0048] Optionally, evaluate whether the fine-tuned control instructions will cause the joint speed or acceleration of the robot arm to exceed its rated range; if the fine-tuning causes the speed or acceleration to be too high, the robot arm may not be able to accurately execute the instructions, or may even cause damage to the mechanical structure, and then such fine-tuning is not feasible.

[0049] In an embodiment of the present invention, by judging whether the control instructions of the robotic arm can be fine-tuned, the control instructions of the robotic arm are flexibly adjusted according to the judgment result, thereby achieving an organic combination of coarse adjustment and fine adjustment. The target person can be transferred to the target position efficiently, safely and accurately, effectively avoiding collisions, ensuring smooth movement, improving rescue efficiency and quality, and at the same time enhancing the adaptability and reliability of the system, reducing the risk of failure, and being suitable for complex and changeable rescue scenarios.

[0050] In some embodiments, step 120 inputs the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model, and obtains a control instruction for the robotic arm output by the decision model, including: Step 121: input the current position and target position of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model; Step 122: Based on the decision model, the motion path of the robotic arm is predicted according to the current posture and target posture of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object, and a collision-free predicted path of the robotic arm is obtained. Based on the collision-free predicted path, the control instructions of the robotic arm are determined and the control instructions of the robotic arm are output.

[0051] It should be noted that the process of predicting the motion path of the robot arm takes into account various possible motion trajectories and the relative position relationship of these trajectories with the target object in space, while combining the kinematic and dynamic characteristics of the robot arm to ensure that the predicted motion path is feasible.

[0052] Optionally, the motion path of the robotic arm is predicted to obtain multiple predicted paths, and a path that can avoid collision between the robotic arm and the target person and the target object is screened out from the multiple predicted paths as the final collision-free predicted path.

[0053] In some embodiments, predicting the motion path of the robotic arm to obtain a collision-free predicted path of the robotic arm includes: The motion path of the manipulator is predicted with the constraint that the predicted minimum distance between the manipulator, the target person, and the target object is greater than a preset distance threshold, and a collision-free predicted path of the manipulator is obtained. The predicted value of the minimum distance is estimated based on the predicted state of the robotic arm, the predicted position of the target person, and the predicted position of the target object.

[0054] Optionally, the preset distance threshold is determined based on user input or according to empirical data.

[0055] It can be understood that by predicting the motion path of the robotic arm with the constraint condition that the predicted value of the minimum distance between the robotic arm and the target person as a whole and the target object is greater than the preset distance threshold, a collision-free predicted path of the robotic arm is obtained. This can effectively avoid collisions between the robotic arm, the target person and the target object during the robotic arm motion planning and control process, thereby ensuring the safe implementation of the rescue mission.

[0056] In some embodiments, optimizing the human-machine tightly coupled system based on the simulation results in step 130 includes: Based on the simulation results, identify abnormal situations and determine the key variables related to the abnormal situations; Conduct risk assessment on the human-machine tightly coupled system based on key variables and obtain risk assessment results; Based on the risk assessment results, the human-machine tightly coupled system is optimized to obtain the optimized human-machine tightly coupled system; Among them, abnormal situations include at least one of the following: The target person does not reach the target posture; The robotic arm collides with the target object while transferring the target person; The target person collides with the target object while being transferred by the robotic arm.

[0057] Among them, key variables include but are not limited to: the current position and target position of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object.

[0058] It is understandable that by simulating and deducing the tightly coupled human-machine system, identifying abnormal situations; determining the key variables related to abnormal situations; and conducting risk assessment based on the key variables to obtain risk assessment results, the key influencing factors and potential risks of system operation can be revealed, so as to provide a reference for the optimal design of complex human-machine systems and the formulation of scientific and reasonable safety management strategies.

[0059] Figure 3 Schematic diagram of the process of training the decision model provided by the embodiment of the present invention. Figure 3 As shown, in some embodiments, the training process of the decision model includes: Step 310: construct an initial decision model, determine a variable set and a rule set of the initial decision model, where the rule set includes a plurality of original inference rules corresponding to the variable set; Step 320: Acquire original sample data, the original sample data including: an original sample pose and an original sample target pose of an original sample target person, an original sample state of an original sample manipulator, and an original sample minimum distance between the original sample manipulator, the original sample target person, and the original sample target object; Step 330: Amplify the original sample data based on generative artificial intelligence technology to obtain sample data; Step 340: Using the sample data as training samples, train the initial decision model. After the training is completed, a decision model is obtained.

[0060] In some embodiments, constructing an initial decision model and determining a variable set and a rule set of the initial decision model include: Obtain experimental data from human subjects in specific task scenarios. The experimental data includes: human operation data, eye movement data, physiological signals reflecting human status, machine state change data, operation videos, and interview data on cognitive decision-making thinking; Based on experimental data, the cognitive architecture is used to formally express people's cognitive decision-making behavior, build an initial decision model, and determine the variable set and rule set of the initial decision model.

[0061] Optionally, recruit a certain number of human subjects and conduct multiple experiments in specific task scenarios to collect human operation data, eye movement data, physiological signals reflecting human status, machine state change data, and operation videos; after the experiment, replay the experimental video, conduct semi-structured interviews on the subjects' operation process, ask the subjects about the cognitive thinking process of making decisions in specific scene fragments, and record the interview data on cognitive decision-making thinking in voice or text form; based on these data, construct an initial decision model and determine the variable set and rule set of the initial decision model.

[0062] It should be noted that in the cognitive model (i.e., the initial decision-making model), cognitive knowledge is divided into declarative knowledge (i.e., the variables and their values required to complete the task) and procedural knowledge (behavioral rules for performing the task). All variables in declarative knowledge can represent the specific state of the system (including people, machines, and environment) after being assigned specific values. The rules of procedural knowledge include conditions and conclusions.

[0063] Specifically, the initial decision model can be formalized as M=<V, P> , V is the variable set, P is the rule set, pt represents the rule activated at the t-th time step, {q0, q1, …, qT} represents the cognitive reasoning path, q0 represents the cognitive reasoning path at the 0th time step, q1 represents the cognitive reasoning path at the 1st time step, and qT represents the cognitive reasoning path at the T-th time step.

[0064] In some embodiments, the original sample data is amplified based on generative artificial intelligence technology to obtain sample data, including: Input the original sample data and noise into the pre-built generative artificial intelligence (GAI) model to obtain new sample data output by the generative artificial intelligence (GAI) model; Aggregate the original sample data and the new sample data to obtain sample data; The training process of the GAI model includes: Obtain training sample data and training noise, determine the training sample data labels corresponding to the training sample data, and the training sample data corresponds to the original sample data; Input the training sample data and training noise into the pre-built initial GAI model to obtain the prediction results of the training sample data output by the initial GAI model; The training sample data labels and the predicted results of the training sample data are compared, and the parameters of the initial GAI model are iteratively optimized to obtain the GAI model.

[0065] Among them, the GAI model learns and analyzes massive amounts of data, understands and masters the rules and patterns behind the data, and thus realizes the simulation and creation of unknown content.

[0066] Optionally, the GAI model uses a large language model (LLM), specifically the LLAMA3-3B model structure.

[0067] Optionally, the new sample data is screened, and reasonable new sample data is aggregated with the original sample data to obtain sample data.

[0068] Optionally, the original sample data and noise are encoded to obtain encoded original sample data and encoded noise, and the encoded original sample data and encoded noise are input into the GAI model to obtain encoded new sample data.

[0069] Optionally, during the GAI model training process, multiple new sample data containing new reasoning paths are obtained, the multiple new sample data containing new reasoning paths are evaluated, and the new sample data containing reasonable new reasoning paths are added to the original sample data.

[0070] In some embodiments, training the initial decision model includes: The sample poses and target poses of the sample target person, the sample state of the sample manipulator, and the sample minimum distance between the sample manipulator, the sample target person, and the sample target object are input into the initial decision model to obtain the predicted control instructions of the manipulator output by the initial decision model; After the robot arm executes the predictive control instruction, the reward value of the predictive control instruction is calculated; According to the reward value of the predicted control instruction, the parameters of the initial decision model are iteratively optimized.

[0071] Optionally, after the robotic arm executes the predictive control instruction, it is determined whether the robotic arm and the target person collide with the target object, and whether the robotic arm transfers the target person to the target posture. Based on the judgment result, the reward value of the predictive control instruction is calculated.

[0072] Figure 4 The following is a flow chart of the modeling and simulation process of the human-machine tightly coupled system provided by the embodiment of the present invention. Figure 4 As shown, in some embodiments, the modeling and simulation deduction process of the human-machine tightly coupled system includes: Step S1: Human-machine integration modeling.

[0073] Step S1-1: Human cognitive behavior modeling.

[0074] Step S1-1-1: Experiment with human subjects. Before the experiment, a robotic arm extravehicular rescue mission scenario was designed. Astronaut 1 was located at the end of the robotic arm and was required to control the movement of the robotic arm within the specified time to transport astronaut 1 to the target location to rescue astronaut 2. During this time, no collision with the space station cabin occurred. The control of the robotic arm included four methods: joint inching, end icon inching, joystick control, and voice control. During the experiment, the experiment was conducted based on the space station robotic arm simulation platform. 85 to 100 people were recruited for the experiment, and the subjects' operation data, eye movement data, robotic arm status data, collision records, etc. were collected. After the experiment, the log files were replayed on the simulation platform, and the subjects were interviewed about their cognitive decision-making process in the specific scenario and the interview data was recorded.

[0075] Step S1-1-2: Establish a preliminary cognitive model (i.e., initial decision-making model).

[0076] Optionally, an Adaptive Control of Thought – Rational (ACT-R) architecture is used to construct an initial decision model.

[0077] Among them, ACT-R is a cognitive architecture that can accurately perform computational modeling of human cognitive processes (such as memory, learning, perception, decision-making, problem solving, etc.). ACT-R provides a unified theoretical framework and simulation tool for understanding how the human brain works.

[0078] Alternatively, cognitive task analysis can be used. First, based on the interview data, the variables involved in the subject's task completion are summarized, such as the difference between the astronaut's current and target positions, and the overall execution strategy is summarized. Next, the specific rules for each step of the cognitive reasoning process under specific conditions are determined, resulting in a variable set V and a rule set P. Rules are expressed in the form of "IF..., THEN..." At the same time, at each reasoning node, other possible reasoning rules not covered by the subject data are appropriately added to the rule set P.

[0079] Optionally, the variable set V and the rule set P are formally represented according to the grammatical form of the ACT-R architecture, where the variable set is called a block and the rule set is called a production; then the reasoning logic of the model is checked and improved to confirm that the model will output reasoning decisions in any state.

[0080] Step S1-1-3: Amplify the subject data.

[0081] Optionally, train a generative artificial intelligence (GAI) model. Specifically, the GAI model uses a large language model (LLM). First, one-hot encode the test data according to the variable set and rule set summarized in S1-1-2, and then train the GAI model using supervised learning.

[0082] Optionally, augment and filter the subject data. Use the trained GAI model to generate a large number of inference paths under different states (variable value combinations). Summarize the newly emerged inference paths, filter out the reasonable ones, and add them to the subject data to obtain the augmented dataset.

[0083] Step S1-1-4: Cognitive model enhancement: Using the augmented dataset, further adjust the ACT-R cognitive model parameters to conform to the distribution of the augmented data.

[0084] Step S1-2: Human-computer interaction modeling.

[0085] Step S1-2-1: Determine the interactive interface variables.

[0086] Step S1-2-2: Determine the human-machine interaction mode. Using the command-based interaction mode, the control instructions output by the decision model are sent to the simulation environment to control the movement of the robotic arm. At the same time, the real-time state parameters of the robotic arm are fed back to the decision model.

[0087] Step S1-2-3: Integrated modeling of the human-machine system. Deploy a spatial robotic arm simulation environment on specific hardware equipment, connect the decision-making model as a participant, and implement integrated modeling of the human-machine system using the interactive method of step S1-2-2.

[0088] Step S2: Human-computer interaction simulation deduction.

[0089] Step S2-1: Human-machine integration simulation. For the robotic arm extravehicular rescue mission, the astronaut's initial and target poses are set. Within these initial states, a large number of simulation experiments are conducted on the decision-making model, and relevant data is collected. Furthermore, the mission status after each interaction step is recorded. For this mission, the status is categorized into three states: "The target pose has not yet been reached, the current step has failed, and the current step has successfully reached the target pose." The initial pose parameters are modified, and the above process is repeated.

[0090] Step S2-2: Analysis of deduction results: The data obtained from the deduction sampling is statistically analyzed according to the changes in the three states with the number of interaction steps.

[0091] Table 1 shows the statistical data from simulations performed in an embodiment of the present invention. As shown in Table 1, every 10 rounds of interaction are grouped together for statistical analysis. 10,116 simulation instances were in the "Not Yet Reached" state ("Not Yet Reached_2") between rounds 10 and 20. After further simulation, 5,656 simulations remained in the "Not Yet Reached" state ("Not Yet Reached_3") between rounds 20 and 30, 4,459 simulations failed ("Failed_3"), and 1 simulation completed the task ("Successful_3"). As can be seen, over 44% of instances failed between rounds 20 and 30, while this percentage did not exceed 20% for other rounds. This indicates that most instances present significant security risks between rounds 20 and 30, providing a reference for system risk warnings.

[0092] Table 1 Simulation data statistics

[0093] The beneficial effects of the present invention are: by introducing a generative artificial intelligence method, it overcomes to a certain extent the problem of insufficient and lacking representative sample size of subject data in traditional human-participated experiments; it proposes a method for integrated modeling and simulation deduction of tightly coupled human-machine systems, providing technical support for the study of human safety issues in complex systems.

[0094] The following describes the human-machine integration simulation and deduction system for robotic arm manipulation provided by an embodiment of the present invention. The human-machine integration simulation and deduction system for robotic arm manipulation described below and the human-machine integration simulation and deduction method for robotic arm manipulation described above can be referenced to each other.

[0095] Figure 5 A schematic diagram of the structure of a human-machine integration simulation system for manipulating a robotic arm is provided in an embodiment of the present invention. Figure 5 As shown, the human-machine integrated simulation and deduction system 500 for manipulating a robotic arm includes: An acquisition unit 510 is configured to acquire the current and target poses of the target person in the human-machine tightly coupled system, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object. The decision unit 520 is configured to input the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model, and obtain control instructions for the robotic arm output by the decision model. The control instructions are used to control the robotic arm to transfer the target person to the target position and avoid collision between the robotic arm, the target person, and the target object during the transfer process. A simulation deduction unit 530 is used to simulate and deduce the human-machine tightly coupled system, obtain simulation results, and optimize the human-machine tightly coupled system based on the simulation results; Among them, the decision model is trained based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: the sample posture and sample target posture of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

[0096] Optionally, the current position and target position of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object are input into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model, including: The current and target poses of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object are input into a pre-built decision model; Based on the decision model, the motion path of the robotic arm is predicted according to the current posture and target posture of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object, and a collision-free predicted path of the robotic arm is obtained. Based on the collision-free predicted path, the control instructions of the robotic arm are determined and output.

[0097] Optionally, predicting the motion path of the manipulator to obtain a collision-free predicted path of the manipulator includes: The motion path of the manipulator is predicted with the constraint that the predicted minimum distance between the manipulator, the target person, and the target object is greater than a preset distance threshold, and a collision-free predicted path of the manipulator is obtained. The predicted value of the minimum distance is estimated based on the predicted state of the robotic arm, the predicted position of the target person, and the predicted position of the target object.

[0098] Optionally, the current state of the robotic arm includes: the current angles of the joints of the robotic arm, the current pitch angle and the current yaw angle of the gimbal on which the robotic arm is located.

[0099] Optionally, the human-machine tightly coupled system is optimized based on the simulation results, including: Based on the simulation results, identify abnormal situations and determine the key variables related to the abnormal situations; Conduct risk assessment on the human-machine tightly coupled system based on key variables and obtain risk assessment results; Based on the risk assessment results, the human-machine tightly coupled system is optimized to obtain the optimized human-machine tightly coupled system; Among them, abnormal situations include at least one of the following: The target person does not reach the target posture; The robotic arm collides with the target object while transferring the target person; The target person collides with the target object while being transferred by the robotic arm.

[0100] Optionally, the training process of the decision model includes: Constructing an initial decision model, determining a variable set and a rule set of the initial decision model, wherein the rule set includes a plurality of original inference rules corresponding to the variable set; Acquire original sample data, the original sample data including: an original sample pose and an original sample target pose of an original sample target person, an original sample state of an original sample manipulator, and an original sample minimum distance between the original sample manipulator and the original sample target person as a whole and an original sample target object; The original sample data is amplified based on generative artificial intelligence technology to obtain sample data; The sample data is used as the training sample to train the initial decision model. After the training is completed, the decision model is obtained.

[0101] Optionally, an initial decision model is constructed, and a variable set and a rule set of the initial decision model are determined, including: Obtain experimental data from human subjects in specific task scenarios. The experimental data includes: human operation data, eye movement data, physiological signals reflecting human status, machine state change data, operation videos, and interview data on cognitive decision-making thinking; Based on experimental data, the cognitive architecture is used to formally express people's cognitive decision-making behavior, build an initial decision model, and determine the variable set and rule set of the initial decision model.

[0102] Optionally, the original sample data is amplified based on generative artificial intelligence technology to obtain sample data, including: Input the original sample data and noise into the pre-built generative artificial intelligence (GAI) model to obtain new sample data output by the generative artificial intelligence (GAI) model; Aggregate the original sample data and the new sample data to obtain sample data; The training process of the GAI model includes: Obtain training sample data and training noise, determine the training sample data labels corresponding to the training sample data, and the training sample data corresponds to the original sample data; Input the training sample data and training noise into the pre-built initial GAI model to obtain the prediction results of the training sample data output by the initial GAI model; The training sample data labels and the predicted results of the training sample data are compared, and the parameters of the initial GAI model are iteratively optimized to obtain the GAI model.

[0103] Optionally, train an initial decision model, including: The sample poses and target poses of the sample target person, the sample state of the sample manipulator, and the sample minimum distance between the sample manipulator, the sample target person, and the sample target object are input into the initial decision model to obtain the predicted control instructions of the manipulator output by the initial decision model; After the robot arm executes the predictive control instruction, the reward value of the predictive control instruction is calculated; According to the reward value of the predicted control instruction, the parameters of the initial decision model are iteratively optimized.

[0104] It should be noted here that the human-machine integration simulation and deduction system for robotic arm manipulation provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned human-machine integration simulation and deduction method embodiment for robotic arm manipulation, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0105] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610 , a communications interface 620 , a memory 630 and a communication bus 640 , wherein the processor 610 , the communications interface 620 and the memory 630 communicate with each other via the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a human-machine integration simulation and deduction method for robotic arm manipulation, which includes: obtaining the current position and target position of the target person in the human-machine tightly coupled system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object; inputting the current position and target position of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model; the control instruction is used to control the robotic arm to transfer the target person to the target position and avoid collision between the robotic arm and the target person and the target object during the transfer process; simulating and deducing the human-machine tightly coupled system to obtain simulation results, and optimizing the human-machine tightly coupled system based on the simulation results; wherein the decision model is obtained by training based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: sample position and sample target position of the sample target person, sample state of the sample robotic arm, and sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

[0106] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0108] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A human-machine integration simulation method for manipulating a robotic arm, characterized in that: include: Obtaining the current and target poses of the target person in the human-machine tightly coupled system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object; The current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object are input into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model; the control instruction is used to control the robotic arm to transfer the target person to the target position and avoid collision between the robotic arm and the target object during the transfer process; Performing simulation on the human-machine tightly coupled system to obtain simulation results, and optimizing the human-machine tightly coupled system based on the simulation results; Among them, the decision model is obtained by training based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: the sample posture and sample target posture of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

2. The human-machine integration simulation method for manipulating a robotic arm according to claim 1, characterized in that: The current position and target position of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object are input into a pre-built decision model to obtain a control instruction for the robotic arm output by the decision model, including: Inputting the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model; Based on the decision model, the motion path of the robotic arm is predicted according to the current posture and target posture of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object, and a collision-free predicted path of the robotic arm is obtained. Based on the collision-free predicted path, the control instructions of the robotic arm are determined and the control instructions of the robotic arm are output.

3. The human-machine integration simulation method for manipulating a robotic arm according to claim 2, characterized in that: The step of predicting the motion path of the robotic arm to obtain a collision-free predicted path of the robotic arm includes: The motion path of the robotic arm is predicted based on the constraint that the predicted minimum distance between the robotic arm, the target person, and the target object is greater than a preset distance threshold, thereby obtaining a collision-free predicted path for the robotic arm. The predicted value of the minimum distance is estimated based on the predicted state of the robotic arm, the predicted posture of the target person, and the predicted posture of the target object.

4. The human-machine integration simulation method for manipulating a robotic arm according to claim 1, characterized in that: The current state of the robotic arm includes: the current angles of the joints of the robotic arm, the current pitch angle and the current yaw angle of the gimbal where the robotic arm is located.

5. The human-machine integration simulation method for manipulating a robotic arm according to claim 1, characterized in that: Optimizing the human-machine tightly coupled system based on the simulation results includes: Based on the simulation results, identifying abnormal conditions and determining key variables related to the abnormal conditions; Performing a risk assessment on the human-machine tightly coupled system based on the key variables to obtain a risk assessment result; Based on the risk assessment result, the human-machine tightly coupled system is optimized to obtain an optimized human-machine tightly coupled system; The abnormal situation includes at least one of the following: The target person fails to reach the target posture; The robotic arm collides with the target object during the process of transferring the target person; The target person collides with the target object while being transferred by the robotic arm.

6. The human-machine integration simulation method for manipulating a robotic arm according to claim 1, characterized in that: The training process of the decision model includes: Constructing an initial decision model, determining a variable set and a rule set of the initial decision model, wherein the rule set includes a plurality of original inference rules corresponding to the variable set; Acquire original sample data, the original sample data including: an original sample pose and an original sample target pose of an original sample target person, an original sample state of an original sample manipulator, and an original sample minimum distance between the original sample manipulator and the original sample target person as a whole and an original sample target object; Amplifying the original sample data based on generative artificial intelligence technology to obtain the sample data; The sample data is used as a training sample to train the initial decision model, and after the training is completed, the decision model is obtained.

7. The human-machine integration simulation method for manipulating a robotic arm according to claim 6, characterized in that: The step of constructing an initial decision model and determining a variable set and a rule set of the initial decision model includes: Acquire experimental data from human subjects in specific task scenarios, including: human operation data, eye movement data, physiological signals reflecting human status, machine state change data, operation videos, and interview data on cognitive decision-making thinking; Based on the experimental data, a cognitive architecture is used to formally express human cognitive decision-making behavior, an initial decision model is constructed, and a variable set and a rule set of the initial decision model are determined.

8. The human-machine integration simulation method for manipulating a robotic arm according to claim 6, characterized in that: The amplifying the original sample data based on the generative artificial intelligence technology to obtain the sample data includes: Inputting the original sample data and noise into a pre-built generative artificial intelligence (GAI) model to obtain new sample data output by the generative artificial intelligence (GAI) model; Aggregating the original sample data and the new sample data to obtain the sample data; The training process of the GAI model includes: Acquire training sample data and training noise, and determine a training sample data label corresponding to the training sample data, wherein the training sample data corresponds to the original sample data; Inputting the training sample data and training noise into a pre-built initial GAI model to obtain a prediction result of the training sample data output by the initial GAI model; The training sample data labels are compared with the prediction results of the training sample data, and the parameters of the initial GAI model are iteratively optimized to obtain the GAI model.

9. The human-machine integration simulation method for manipulating a robotic arm according to claim 6, characterized in that: The training of the initial decision model comprises: Inputting the sample pose and sample target pose of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm, the sample target person, and the sample target object into the initial decision model, and obtaining a predictive control instruction for the robotic arm output by the initial decision model; After the robotic arm executes the predictive control instruction, calculating a reward value of the predictive control instruction; Iteratively optimize the parameters of the initial decision model according to the reward value of the predictive control instruction.

10. A human-machine integrated simulation and deduction system for manipulating a robotic arm, characterized in that: include: an acquisition unit, configured to acquire the current posture and target posture of the target person in the human-machine tight coupling system, the current state of the robotic arm, and the current minimum distance between the robotic arm and the target person as a whole and the target object; a decision unit, configured to input the current and target positions of the target person, the current state of the robotic arm, and the current minimum distance between the robotic arm, the target person, and the target object into a pre-built decision model, and obtain a control instruction for the robotic arm output by the decision model; the control instruction is used to control the robotic arm to transfer the target person to the target position and to prevent the robotic arm, the target person, and the target object from colliding during the transfer process; a simulation deduction unit, configured to simulate and deduce the human-machine tightly coupled system to obtain simulation results, and optimize the human-machine tightly coupled system based on the simulation results; Among them, the decision model is obtained by training based on sample data; the sample data is obtained by amplifying the original sample data based on generative artificial intelligence technology; the sample data includes: the sample posture and sample target posture of the sample target person, the sample state of the sample robotic arm, and the sample minimum distance between the sample robotic arm and the sample target person as a whole and the sample target object.

Images