A process knowledge and interactive semantic driven active human-machine collaboration method

Abstract

The application provides a process knowledge and interactive semantic driven active man-machine cooperation method, an initial generation result is obtained by gradually optimizing and updating general task description text and dynamic assembly task interactive description text in the manufacturing field, the final generation result is obtained by evaluating according to semantic similarity classification standards and real labels, observation sampling of posture sequences of an operator when implementing assembly behaviors according to the generation result is performed to obtain observation postures and dynamic scene semantic information, a diffusion matrix is obtained by extracting features, an inherent matrix and a weight matrix are further processed to form a point matrix feature, the point matrix feature is mapped into a Gaussian distribution coding set feature map, a current posture sequence is obtained by mapping a latent coding factor sampled into the posture sequence, an operator assembly behavior prediction result is obtained by prediction, an agent action and the operator assembly behavior are evaluated in combination with an assembly environment state and an assembly task state, a current state and a reward function feedback are provided to the agent, and robot movement is controlled.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field, and in particular to an active human-robot collaboration method driven by process knowledge and interactive semantics. BACKGROUND

[0002] Under the background of industry 5.0, with the breakthrough of artificial intelligence technologies such as pre-training large models, personalized product human-robot collaboration assembly has attracted great attention, and active human-robot collaboration has made great progress in perception, decision-making and other aspects. As an important research direction of human-based manufacturing, active human-robot collaboration emphasizes the active human-robot collaboration that enhances the task execution ability through perception, reasoning and environmental interaction. It is not only to form intelligence through calculation, but more importantly, to generate knowledge in dynamic interaction with the environment, thereby shaping active intelligent ability.

[0003] At present, the human-robot collaboration assembly of personalized products mainly faces the following challenges:

[0004] 1. There is a large amount of dynamic interaction information in the human-robot collaboration assembly of personalized products, and the combination of process knowledge and interactive semantics needs to be considered. This process has high dynamicity and uncertainty, and requires robots to efficiently adapt to the rapid changes of operator behavior, component condition and working environment.

[0005] 2. The diversity and complexity of personalized products increase the factors that operators need to consider when selecting assembly paths, which may hinder task progress, highlighting the need for effective communication and information exchange between operators and robots.

[0006] 3. The assembly process of personalized products has unstructured environmental uncertainty, and operator behavior will change accordingly. The human-robot collaboration process is dynamic, and compared with structured environments with fixed distribution and rules, the assembly space has unstructured characteristics. Information fusion and decision support are crucial, and robots must quickly process operator instructions and environmental feedback to make accurate decisions in dynamic environments, resulting in new challenges for robots in spatial perception, task planning and reasoning. SUMMARY

[0007] The technical scheme of the application aims to provide an active human-robot collaboration method driven by process knowledge and interactive semantics.

[0008] The technical scheme of the application provides an active human-robot collaboration method driven by process knowledge and interactive semantics, including the following steps:

[0009] A general manufacturing dataset capable of enhancing the understanding of general manufacturing tasks is made, including general manufacturing field pictures and corresponding manufacturing field general task description texts;

[0010] A specific domain manufacturing dataset capable of enhancing the understanding of dynamic assembly tasks is made, including: assembly scene pictures corresponding to dynamic assembly tasks and interactive description texts of corresponding dynamic assembly tasks;

[0011] Using process knowledge and interactive semantics, the manufacturing domain general task description text and the dynamic assembly task interactive description text are updated step by step in the linear projection layer and the low-rank adaptive layer to obtain an initial generated result. According to the semantic similarity classification standard, the semantic similarity score between the generated result and the real label is evaluated using GPT-4. When the semantic similarity score is lower than the preset threshold, it is determined that the initial generated result does not match the real label, and the generated result is updated step by step to obtain the final generated result that matches the real label in semantics.

[0012] The posture sequence of the operator when implementing the assembly behavior according to the generated result is observed and sampled to obtain observation postures and dynamic scene semantic information. A semantic smoothing enhanced spatio-temporal graph convolution network with residual connection is used to extract features to obtain a diffusion matrix.

[0013] An intrinsic matrix is obtained from the sampling distribution of the observation postures. A weight matrix is obtained by sampling from a uniform distribution and applying a Gumbel distribution using a Softmax sampling method.

[0014] The intrinsic matrix and the diffusion matrix form a diffusion space together. The intrinsic matrix is linearly combined in this space according to the weight matrix to generate multiple points. The intrinsic matrix and the weight matrix are multiplied to form a point matrix. An MLP is used to learn to map the features in the point matrix to a set of Gaussian distribution encoding set feature maps with diverse latent encoding. A diverse latent encoding factor is sampled from the Gaussian distribution encoding set feature map. The latent encoding factor is mapped to the posture sequence to obtain the current posture sequence. The dynamic scene semantic information and the latent encoding factor are fused into the observed posture to predict the diversity of the operator's assembly behavior and obtain the operator's assembly behavior prediction result.

[0015] According to the assembly environment state and the assembly task state, the operator's assembly behavior prediction result is combined with the task planning. A reward function is established as a Markov decision process to evaluate the agent's action and the operator's assembly behavior, and to provide feedback to the agent about the current state and the reward function. The assembly operation knowledge is accumulated to realize interactive knowledge continuous learning and optimization decision, and to control the robot motion.

[0016] Preferably, the semantic similarity score formula is:

[0017]

[0018] In the formula, n represents the classification level in the semantic similarity standard, Number nrepresent all the generated results that meet the nth level.

[0019] Preferably, in the operator assembly behavior diversified prediction of the operator assembly behavior prediction result, the future posture sequence in the operator assembly behavior prediction result is recursively smoothed and refined, the historical posture sequence remains unchanged in the initial state, and a supervised sequence is generated for feedback adjustment.

[0020] Preferably, the reward function:

[0021]

[0022] In the formula, s or represents the distance between the operator and the robot, s op represents the distance between the operator and the component, s rp represents the distance between the robot and the component, n t represents the total assembly task to be completed, and cst represents whether the current assembly task is completed, n st represents the number of completed assembly tasks.

[0023] The technical scheme of the present application provides an active human-robot collaboration method driven by process knowledge and interactive semantics, which obtains an initial generated result by gradually optimizing and updating general task description texts in the manufacturing field and dynamic assembly task interactive description texts, evaluates the final generated result according to a semantic similarity classification standard and a real label, observes and samples the posture sequence of the operator when implementing the assembly behavior according to the generated result to obtain observed postures and dynamic scene semantic information, extracts features to obtain a diffusion matrix, further processes to obtain an intrinsic matrix and a weight matrix to form a point matrix feature, maps to a Gaussian distribution coding set feature map, samples a latent coding factor to map to a posture sequence to obtain a current posture sequence, predicts an operator assembly behavior prediction result, evaluates the agent action and the operator assembly behavior in combination with the assembly environment state and the assembly task state, provides the current state and a reward function feedback to the agent, and controls the robot motion. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A general framework schematic diagram of an active human-robot collaboration method driven by process knowledge and interactive semantics is provided for the embodiment of the present application.

[0025] Figure 2 A low-rank adaptive fine-tuning process schematic diagram in an active human-robot collaboration method driven by process knowledge and interactive semantics is provided for the embodiment of the present application.

[0026] Figure 3 An operator behavior diversified prediction process schematic diagram in an active human-robot collaboration method driven by process knowledge and interactive semantics is provided for the embodiment of the present application.

[0027] Figure 4 A schematic diagram of a robot motion adaptive decision-making process in a process knowledge and interactive semantic driven active human-robot collaboration method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described with reference to the following specific examples. It should be understood that these examples are intended to illustrate the present application and are not intended to limit the scope of the present application. Furthermore, it should be understood that various modifications and changes can be made to the present application by those skilled in the art upon reading the contents of the present application, and such equivalent forms are also within the scope of the appended claims.

[0029] As shown in Figure 1 An embodiment of the present application provides a process knowledge and interactive semantic driven active human-robot collaboration method, which comprises:

[0030] As shown in Figure 2 A general manufacturing dataset capable of enhancing the understanding of general manufacturing tasks is made, which comprises general manufacturing field pictures and corresponding manufacturing field general task description texts, and a manufacturing task multimodal prompt word is designed in a fine-tuning process by combining the embedding features of the manufacturing field pictures and the corresponding language instructions of the manufacturing field general task description texts, so as to provide a standardized input format. Specifically, it is expressed as: [picture]<general manufacturing picture>[description]<picture description>[prompt instruction]<instruction>, for example, the picture description is "an operator stands beside a reducer without a cover" and the corresponding picture, and a prompt word such as "please briefly describe this picture" is used to guide the learning of the mapping relationship between the general manufacturing picture and the description text by combining the visual features and the language prompts, so as to provide a basis for adapting to different specific tasks.

[0031] A specific field manufacturing dataset capable of enhancing the understanding of dynamic assembly tasks is made, including: assembly scene pictures corresponding to dynamic assembly tasks and interactive description texts corresponding to dynamic assembly tasks, combined with the embedding features of the assembly scene pictures and the language instructions corresponding to the interactive description texts of the dynamic assembly tasks, a dynamic assembly task multi-modal prompt word is designed in the fine-tuning process. Specifically represented as: [Picture]<Assembly scene picture>[Description]<Dynamic interactive description>[Prompt instruction]<Instruction>, such as the text "The current operator is holding the two ends of the gear shaft with both hands, and slightly bending over in front of the reducer, his position is close to the third slot of the reducer, so it is predicted that the operator will move the gear shaft horizontally from top to bottom, and put it into the third slot of the reducer" describes the assembly scene in the corresponding picture, and predicts the future dynamic process, with a prompt word like "According to the given assembly scene picture, please describe the current scene as much as possible, and predict the future dynamic process", guide to learn the mapping relationship between the assembly scene picture and the dynamic interactive description text through the combination of visual features and language prompts, realize the understanding of the current assembly scene and the preliminary planning of the future assembly.

[0032] Using process knowledge and interactive semantics, the manufacturing field general task description text and the dynamic assembly task interactive description text are updated step by step in the linear projection layer and the low-rank adaptive layer, imitating human learning thinking, and gradually transitioning from general tasks to specific tasks, obtaining initial generated results, using GPT-4 according to semantic similarity classification standards to evaluate the semantic similarity score between the generated results and the true labels, when the semantic similarity score is lower than the preset threshold, it is determined that the initial generated result does not match the true label, and the generated result is updated step by step to form a closed loop, and finally the final generated result that matches the semantic of the true label is obtained.

[0033] In order to facilitate GPT-4 to evaluate the semantic similarity of the text more finely, a semantic similarity classification standard is designed to realize the qualitative evaluation of the semantic similarity between the generated results and the true labels. At the same time, artificial experience is introduced to verify the evaluation results of GPT-4, ensure the accuracy of the generated results, guarantee the accurate understanding of human-computer collaboration knowledge, and realize the learning and optimization of specific field human-computer collaboration interactive knowledge.

[0034] The semantic similarity score represents the qualitative evaluation of semantic similarity, which is as follows:

[0035]

[0036] In the formula, n represents the classification level in the semantic similarity standard, Number n represents all generated results that meet the nth level, and this weighted average score can better reflect the overall quality of the generated results in terms of semantic similarity.

[0037] like Figure 3 As shown, the observed posture X∈R is obtained by observing and sampling the posture sequence of the operator when performing assembly actions based on the generated results. S×C×L Based on the semantic information of the dynamic scene, a semantically smoothed spatiotemporal graph convolutional network with residual connections is used to extract features and obtain the diffusion matrix D∈R. S×D_D Where S represents the number of skeletal joints, and D_D is the dimension of the feature; from the observed pose X∈R S×C×L The intrinsic matrix I∈R is obtained from the sampling distribution. M×D_I The weight matrix W∈R is obtained by sampling from a uniform distribution and normalizing it using the Gumbel distribution and the Softmax sampling method. D_W×M ; Set the intrinsic matrix I∈R M ×D_I With weight matrix W∈R D_W×M Perform multiplication to form a point matrix P∈R D_W×D_I MLP learning is used to transform the point matrix P∈R D _W×D_I The features in the image are mapped to a set of Gaussian distributed coding feature maps with diverse latent codes. Diverse latent coding factors are sampled from the Gaussian coding feature maps and mapped to the pose sequence to obtain the current pose sequence. In other words, the dynamic scene semantic information and latent coding factors are fused into the observed pose to perform diversified prediction of operator assembly behavior and obtain the operator assembly behavior prediction result. This ensures that the current pose has a basis for diversified prediction and generates a variety of high-precision potential behaviors for dynamic human movement.

[0038] The intrinsic matrix I∈R M×D_I Each row contains a D_I-dimensional intrinsic vector, the total number of intrinsic vectors is M, and the intrinsic matrix I∈R M×D_I With diffusion matrix D∈R S×D_D The fusion of these elements together forms the diffusion space, with the intrinsic matrix I∈R M ×D_I According to the weight matrix W∈R D_W×M Multiple points are generated in this space through linear combination. These points are used to generate diverse distributions, with a weight matrix W∈R. D_W×M It has M weights with a D_W dimension.

[0039] In the Gaussian distribution coding set, each row of the point matrix corresponds to a point of D_I dimension and a point of D_W dimension originating from the diffusion space.

[0040] This step enhances the diversity of inherent weights derived from the observation pose by combining weights derived from the diffusion matrix and the Gumbel distribution, ensuring that the motion distribution diffuses as much as possible while maintaining its inherent pose.

[0041] The semantic smoothness enhanced spatio-temporal graph convolution prediction network is composed of a multi-step structure, and each step includes an encoder-decoder module.

[0042] In the operator assembly behavior diversification prediction, the future pose sequence SF=(SF1, SF2, …, SFt) in the operator assembly behavior prediction result is recursively smoothed, and the future pose sequence SF is gradually refined, while the historical pose sequence SH remains unchanged in the initial state, to generate a supervised sequence SHFt-1, SHFt-2, …, SHF1 to adjust the corresponding historical pose SH=(SH1, SH2, …, SHt), and build a feedback mechanism. The specific definition is as follows:

[0043]

[0044] Where i-1 represents the time of the last historical pose sequence.

[0045] As shown in Figure 4 According to the assembly environment state and the assembly task state, the reward function is established as a Markov decision process to evaluate the agent action and the operator assembly behavior, and feedback about the current state and the reward function is provided to the agent, and assembly operation knowledge is accumulated, so as to realize interactive knowledge continuous learning and optimization decision, control the robot motion, and ensure the understanding and generation ability.

[0046] The reward function is crucial for the agent to learn correct decisions, because it provides feedback to the agent to help decision-making, and the reward function ensures the timeliness and safety of the robot decision-making. For decision execution, the reward function considers minimizing the robot decision time and maximizing human satisfaction at the same time. The reward function is as follows:

[0047]

[0048] In the formula, s or represents the distance between the operator and the robot, s op represents the distance between the operator and the component; s rp represents the distance between the robot and the component; n t represents the total assembly task to be completed; cst represents whether the current assembly task is completed, 1 for completion and -1 for non-completion; n st represents the number of completed assembly tasks. The above reward function can guide the robot to continuously learn decision-making behavior under different spatial distances, and ensure that the final decision meets the requirements of shorter time and larger safety distance.

[0049] The embodiment of the application provides a process knowledge and interactive semantic driven active human-machine cooperation method, which has the following beneficial effects:

[0050] 1. For the process dynamics and human-machine cooperation problems in human-machine cooperation assembly, from the aspects of spatial perception, process analysis, decision reasoning and task execution, an active human-machine cooperation method enhanced by personalized product process knowledge and interactive semantics is proposed.

[0051] 2. For the problem that static domain knowledge cannot make large models understand dynamic interaction in human-machine cooperation assembly, a process knowledge and interactive semantic enhanced low-rank adaptive fine-tuning method is proposed, which uses data representing assembly dynamic interaction to update only the linear projection layer and low-rank adaptive layer of the large model, forming a specific domain large model.

[0052] 3. For the problem that the lexical similarity evaluation method cannot meet the openness of the generated results of the large model, a semantic similarity driven fine-tuning evaluation and optimization method is proposed, which introduces GPT-4 and human experience to evaluate the generated results. The results that do not meet the requirements will be regenerated, forming a closed loop to ensure the understanding of the large model to the domain knowledge.

[0053] 4. For the problem that highly random operator assembly behavior will interfere with robot embodiment reasoning, human factors are introduced into the human-machine cooperation loop, and an operator intention diversification prediction method is proposed to improve the ability of the large model to predict operator behavior. Through "human-in-the-loop", continuous improvement of human-machine cooperation is carried out to ensure task execution.

[0054] 5. For the deficiency of robot motion control, considering that traditional methods such as expert experience have certain subjectivity, a deep reinforcement learning driven robot motion adaptive decision method is proposed to continuously learn and optimize operator intention prediction and scene interaction knowledge, and continuously improve the task reasoning and adaptability to dynamic scenes.

Claims

1. A proactive human-machine collaboration method driven by process knowledge and interactive semantics, characterized in that, Includes the following steps: To create a general manufacturing dataset that can enhance the understanding of general manufacturing tasks, including: images of general manufacturing domains and corresponding textual descriptions of general manufacturing tasks; Create a domain-specific manufacturing dataset that enhances the understanding of dynamic assembly tasks, including: assembly scene images corresponding to dynamic assembly tasks and corresponding interactive description text of dynamic assembly tasks; By leveraging process knowledge and interactive semantics, the general task description text and dynamic assembly task interactive description text in the linear projection layer and low-rank adaptive layer are progressively optimized and updated to obtain the initial generation result. Based on the semantic similarity classification standard, GPT-4 is used to evaluate the semantic similarity score between the generated result and the real label. When the semantic similarity score is lower than the preset threshold, it is determined that the initial generation result does not match the real label. The generation result is then progressively optimized and updated again to finally obtain the final generation result that semantically matches the real label. The operator's posture sequence when performing assembly behavior based on the generated results is observed and sampled to obtain observed posture and dynamic scene semantic information. The diffusion matrix is ​​obtained by extracting features using a semantic smoothing-enhanced spatiotemporal graph convolutional network with residual connections. The intrinsic matrix is ​​obtained from the sampling distribution of the observed attitude. The weight matrix is ​​obtained by sampling from the uniform distribution and normalizing it using the Softmax sampling method with the Gumbel distribution. The fusion of the intrinsic matrix and the diffusion matrix forms the diffusion space. The intrinsic matrix generates multiple points in this space through linear combination with the weight matrix. The intrinsic matrix and the weight matrix are multiplied to form a point matrix. MLP learning is used to map the features in the point matrix to a set of Gaussian distributed coding feature maps with diverse latent codes. Diverse latent coding factors are sampled from the Gaussian coding feature maps and mapped to the pose sequence to obtain the current pose sequence. This realizes the fusion of dynamic scene semantic information and latent coding factors into the observed pose, and performs diversified prediction of operator assembly behavior to obtain the operator assembly behavior prediction result. Based on the assembly environment state and assembly task state, and combined with the prediction results of the operator's assembly behavior in the task planning, a Markov decision process is established using a reward function to evaluate the agent's actions and the operator's assembly behavior. The agent is then provided with feedback on the current state and the reward function, accumulating assembly operation knowledge. This enables continuous learning of interactive knowledge and optimization of decision-making to control the robot's movement.

2. The proactive human-machine collaboration method driven by process knowledge and interactive semantics as described in claim 1, characterized in that, The semantic similarity scoring formula is as follows: In the formula, n represents the classification level in the semantic similarity criterion, and Number n This represents all generated results that match the nth level.

3. The proactive human-machine collaboration method driven by process knowledge and interactive semantics as described in claim 1, characterized in that, In the operator assembly behavior prediction results obtained from the operator assembly behavior diversification prediction, the future posture sequence in the operator assembly behavior prediction results is recursively smoothed and refined, while the historical posture sequence remains unchanged in the initial state, and a supervision sequence is generated for feedback adjustment.

Images