Dynamic man-machine cooperation safety area prediction method and device, medium and product
Through the combination of space-time graph convolution network and deep learning model, the safe area of the robot arm is predicted in real time, which solves the problem of inaccurate definition of safe area in the collaborative operation of human-machine in the existing technology, and realizes intelligent collision avoidance control between the robot arm and operators, improving the safety and flexibility of the operation.
Patent Information
- Application Number
- CN202510360611.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-25
- Publication Date
- 2025-07-08
AI Technical Summary
The existing technology cannot predict the future motion trajectory and behavioral patterns of operators in real time and accurately, resulting in inaccurate definition of safety areas in man-machine collaborative operations in complex environments, lacking intelligent collision avoidance capabilities and adaptability, and traditional methods have insufficient flexibility and safety.
The space-time graph convolution network is used to combine deep learning models to obtain the motion data of operators and robotic arms in real time, predict the safe area through spatiotemporal feature analysis, and use dynamic planning methods to plan the optimal motion trajectory of the robotic arms to realize real-time prediction and collision avoidance control of the dynamic safety area.
It improves the accuracy and real-timeness of safety area prediction, realizes intelligent collision avoidance control between the robot arm and the operator, and improves the safety and continuity of human-machine collaborative operations.
Smart Images
Figure CN120269549A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of automation control, and particularly to a method, device, medium, and product for predicting a dynamic human-machine collaboration safety area. Background Art
[0002] With the rapid development of automation technology and robotics technology, human-machine collaboration systems have been widely used in various industrial production and high-risk operation environments. Especially in the scenario where a robotic arm and an operator work together, how to ensure that the robotic arm can work safely and efficiently with the operator in a dynamic environment has become a key problem to be solved urgently.
[0003] Traditional safety protection means mainly rely on the fixed safety boundary of the robotic arm or simple static obstacle avoidance control. Although these methods can reduce the occurrence of safety accidents to a certain extent, they have significant limitations.
[0004] In the prior art, many human-machine collaboration systems adopt sensor-based safety detection and control methods. These methods monitor the position and movement state of the operator by installing sensors, and give a warning or stop the operation when the robotic arm approaches the operator. However, these technologies usually cannot accurately predict the future movement trajectory and behavior pattern of the operator, resulting in a relatively rough definition of the safety area and being unable to adapt to complex dynamic operation environments. In addition, most of the prior art fails to make full use of the spatio-temporal dynamic relationship between the operator and the robotic arm, lacking intelligent collision avoidance ability and adaptability.
[0005] Specifically, most traditional safety area prediction methods rely on fixed algorithms or threshold settings, and cannot be dynamically adjusted in real time according to the movement state and behavior of the operator. The path planning and obstacle avoidance control of the robotic arm usually adopt simple rules or static models, which cannot respond to the changes of the safety area in real time and are also difficult to make flexible path planning according to the actual operation requirements. Especially in complex working scenarios, traditional methods often lead to inflexible operation of the robotic arm and low safety protection efficiency.
[0006] In addition, although deep learning and intelligent control methods have made significant progress in some fields, in terms of spatio-temporal data processing and real-time safety area prediction, the prior art has not been able to effectively integrate spatio-temporal features and human-machine collaboration dynamics, lacking an efficient and safe control strategy that can combine operator behavior prediction and robotic arm movement collaboration.
[0007] Therefore, based on the above problems, there is an urgent need to provide a method for predicting a dynamic human-machine collaboration safety area, which can comprehensively consider the spatio-temporal dynamic relationship between the operator and the robotic arm, realize real-time and safe dynamic safety area prediction and collision avoidance control, so as to meet the requirements of human-machine collaboration operations in complex environments. Summary of the Invention
[0008] The purpose of this application is to provide a dynamic human - machine collaborative safety area prediction method, device, medium and product, which can realize dynamic safety area prediction and collision avoidance control in real - time and safely.
[0009] To achieve the above - mentioned purpose, this application provides the following solutions:
[0010] In the first aspect, this application provides a dynamic human - machine collaborative safety area prediction method, and the dynamic human - machine collaborative safety area prediction method includes:
[0011] Obtain the operator's motion data and the robotic arm's motion data in real - time respectively; the operator's motion data includes: the three - dimensional spatial position coordinates, motion speed, and acceleration of the operator; the robotic arm's motion data includes: the joint angles and the end - effector pose of the robotic arm.
[0012] According to the operator's motion data and the robotic arm's motion data, use a spatio - temporal graph convolutional network to determine spatio - temporal features.
[0013] According to the spatio - temporal features, use a deep learning model to predict the safety area of the robotic arm.
[0014] According to the prediction result of the real - time safety area, use the dynamic programming method to plan the optimal motion trajectory of the robotic arm.
[0015] Optionally, the step of obtaining the operator's motion data and the robotic arm's motion data in real - time respectively specifically includes:
[0016] Use a vision sensing system to obtain the operator's motion data in real - time; the vision sensing system is used to obtain the real - time image of the operator by using a camera; and according to the real - time image, use a human skeleton detection algorithm to determine the operator's motion data.
[0017] Use a vision sensing system to obtain the robotic arm's motion data in real - time.
[0018] According to the operator's motion data and the robotic arm's motion data, construct a dynamic dataset of the human - machine collaborative scenario.
[0019] Optionally, the step of using a spatio - temporal graph convolutional network to determine spatio - temporal features according to the operator's motion data and the robotic arm's motion data specifically includes:
[0020] According to the operator's motion data, use the temporal convolutional layer in the spatio - temporal graph convolutional network to determine the time series of the operator's motion features; and determine the features after temporal convolution according to the time series of the operator's motion features.
[0021] Determine the adjacency matrix according to the robotic arm's motion data and the operator's motion data.
[0022] According to the adjacency matrix and the features after temporal convolution, the graph convolution layer in the spatio-temporal graph convolutional network is used to determine the spatio-temporal features.
[0023] Optionally, the deep learning model is a long short-term memory network.
[0024] Optionally, according to the prediction result of the real-time safety area, using the dynamic programming method to plan the optimal motion trajectory of the robotic arm, specifically including:
[0025] Determine the safety area constraint according to the prediction result of the real-time safety area;
[0026] Determine the constraint condition according to the safety area constraint;
[0027] Use the dynamic programming method to plan the optimal motion trajectory of the robotic arm according to the constraint condition.
[0028] Optionally, the determining the safety area constraint according to the prediction result of the real-time safety area specifically includes:
[0029] Using the formula ||T m -S t ||2≥d safe To determine the safety area constraint;
[0030] Where, T m is the position of the end of the robotic arm, S t is the boundary of the safety area at time t, d safe is the safety distance threshold, and || ||2 is the L2 norm.
[0031] Optionally, the determining the constraint condition according to the safety area constraint specifically includes:
[0032] Using the formula To determine the constraint condition;
[0033] Where, t0 is the starting time of trajectory planning, t n is the ending time of trajectory planning, T m (t) is the position of the end of the robotic arm at time t, || || 2 is the square of the norm.
[0034] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the dynamic human-machine collaborative safety area prediction method.
[0035] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the dynamic human-machine collaborative safety area prediction method described above is implemented.
[0036] In a fourth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the dynamic human-machine collaborative safety area prediction method described above is implemented.
[0037] According to the specific embodiments provided by the present application, the present application has the following technical effects:
[0038] The present application provides a dynamic human-machine collaborative safety area prediction method, device, medium and product. By using spatio-temporal graph convolutional network based on the operator's motion data and the robotic arm's motion data, spatio-temporal features are determined, that is, by combining spatio-temporal graph convolutional network (ST-GCN), the spatio-temporal dynamic relationship between the operator and the robotic arm is comprehensively considered; according to the spatio-temporal features, combined with a deep learning model, the prediction of the dynamic safety area is carried out, and the safety distance between the robotic arm and the operator is ensured through adaptive collision avoidance control, so as to achieve intelligent collision avoidance. The present application is applied to a human-machine collaborative working environment. By real-time monitoring and predicting the motion state of the operator, the safety area boundary of the robotic arm is calculated and predicted in real time, so that the robotic arm can operate within the dynamic safety area, thus realizing dynamic safety area prediction and collision avoidance control in real time and safely to meet the requirements of human-machine collaborative operation in complex environments. Description of the Drawings
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0040] Figure 1 It is a schematic flowchart of a dynamic human-machine collaborative safety area prediction method in an embodiment of the present application;
[0041] Figure 2 It is a schematic diagram of the principle of a dynamic human-machine collaborative safety area prediction method in an embodiment of the present application. Detailed Embodiments
[0042] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.
[0043] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0044] In an exemplary embodiment, as Figure 1 and Figure 2 shown, a dynamic human-machine collaborative safety area prediction method is provided, and this method includes the following S101 to S104. Among them:
[0045] S101, respectively and real-time obtain the operator's motion data and the robotic arm's motion data; the operator's motion data includes: the three-dimensional spatial position coordinates P p =(x p , y p , z p ), the motion speed vector V p =(v x , v y , v z ), and the acceleration vector A p =(a x , a y , a z ); the robotic arm's motion data includes: the joint angles Q m =(q1, q2,..., q n ) and the end pose T m =(x m , y m , z m , θ m );
[0046] S101 specifically includes:
[0047] S11, use a vision sensing system to real-time obtain the operator's motion data; the vision sensing system is used to obtain the real-time image of the operator by using a camera; and according to the real-time image, use a human skeleton detection algorithm to determine the operator's motion data;
[0048] S12, use a vision sensing system to real-time obtain the robotic arm's motion data;
[0049] S13, according to the operator's motion data and the robotic arm's motion data, construct a dynamic data set D for the human-machine collaborative scenariot = {P p (t), V p (t), A p (t), Q m (t), T m (t)}.
[0050] Among them, the human motion data further includes: other biological motion characteristics; such as joint angle changes. The robotic arm motion data further includes: control states such as speed and acceleration;
[0051] As Figure 2 shown, the dynamic dataset of the human-machine collaboration scenario in this application is multi-source data.
[0052] S102. According to the operator's motion data and the robotic arm's motion data, use a spatio-temporal graph convolutional network to determine spatio-temporal features;
[0053] S102 specifically includes:
[0054] S21. According to the operator's motion data, use the temporal convolutional layer in the spatio-temporal graph convolutional network to determine the time series of the operator's motion features; and determine the feature H after temporal convolution according to the time series of the operator's motion features t = Conv t (X p ) = X p * w t ; the convolutional kernel of the temporal convolutional layer is w t ; X p is the time series of the operator's motion features;
[0055] S22. Determine the adjacency matrix A according to the robotic arm's motion data and the operator's motion data; the adjacency matrix A represents the relationship between nodes;
[0056] S23. According to the adjacency matrix A and the feature H after temporal convolution t , use the graph convolutional layer in the spatio-temporal graph convolutional network to determine the spatio-temporal feature H g = GCN(H t , A) = σ(AH t W). Among them, W is the learned weight matrix, and σ is the activation function. The spatio-temporal feature H g is the joint feature in the time and space dimensions.
[0057] In the graph convolutional layer, graph convolution is used to extract the spatial relationship between the operator and the robotic arm. Assume that G = (V, E) is a graph structure, where V represents the node set (operator and robotic arm), and E represents the edge between nodes (human-machine collaboration relationship).
[0058] S103. Predict the safe area of the robotic arm using a deep learning model according to spatio-temporal characteristics. The deep learning model can predict the boundary of the safe area at future moments by learning the mapping relationship between the movement patterns of the operator and the safe area.
[0059] Assume that the boundary of the safe area can be represented by a multi-dimensional function S(t), which dynamically calculates the boundary of the safe area based on the movement state of the operator. The output of the deep learning model is the boundary coordinates S t =(x s , y s , z s ), and this boundary changes continuously with the movement states of the operator and the robotic arm.
[0060] When the deep learning model is a long short-term memory network (LSTM), the prediction of the boundary coordinates is:
[0061] S t+1 = f(S t , H g (t));
[0062] where f() is the prediction function, and H g (t) is the spatio-temporal characteristic at time t.
[0063] S104. Plan the optimal motion trajectory of the robotic arm using the dynamic programming method according to the prediction result of the real-time safe area.
[0064] S104 specifically includes:
[0065] S41. Determine the safe area constraint according to the prediction result of the real-time safe area;
[0066] Use the formula ||T m - S t ||2 ≥ d safe to determine the safe area constraint;
[0067] where T m is the position of the end of the robotic arm, S t is the boundary of the safe area at time t, d safe is the safe distance threshold, and || ||2 is the L2 norm.
[0068] Among them, the position of the end of the robotic arm is calculated using the forward kinematics method.
[0069] S42. Determine the constraint conditions according to the safe area constraint;
[0070] Use the formula to determine the constraint conditions;
[0071] where \(t_0\) is the starting time of trajectory planning, \(t\) n is the ending time of trajectory planning, \(T\) m (t) is the position of the end effector of the robotic arm at time \(t\), \(\|\ \|\) 2 is the square of the norm.
[0072] S43. Use the dynamic programming method to plan the optimal motion trajectory of the robotic arm according to the constraint conditions.
[0073] This application constructs a human - robot collaborative scenario modeling method based on spatio - temporal graph convolutional network, improving the accuracy and real - time performance of safety area prediction; realizing the intelligent perception of the operator's motion intention, enabling the robotic arm to predictively adjust its safety area; developing a collision avoidance control strategy under dynamic safety area constraints, significantly enhancing the safety and continuity of human - robot collaborative operations.
[0074] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non - volatile storage medium and an internal memory. The non - volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non - volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a dynamic human - robot collaborative safety area prediction method.
[0075] In an exemplary embodiment, a computer - readable storage medium is provided, storing a computer program, which when executed by a processor, implements the steps in the above - mentioned method embodiments.
[0076] In an exemplary embodiment, a computer program product is provided, including a computer program, which when executed by a processor, implements the steps in the above - mentioned method embodiments.
[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.
[0078] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0079] The databases involved in the various embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the various embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.
[0080] In the present application, all actions of obtaining signals, information, or data are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where it is located and obtaining authorization from the owner of the corresponding device.
[0081] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.
[0082] In this article, specific examples are used to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. At the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. To sum up, the content of this specification should not be construed as a limitation to this application.
Claims
1. A dynamic human-machine collaborative safety area prediction method, characterized in that The described dynamic human-machine collaborative safety area prediction method includes: Obtaining the operator's motion data and the robotic arm's motion data in real time respectively; the operator's motion data includes: the three-dimensional spatial position coordinates, motion speed, and acceleration of the operator; the robotic arm's motion data includes: the joint angles and end poses of the robotic arm. Based on the operator's motion data and the robotic arm's motion data, use a spatio-temporal graph convolutional network to determine spatio-temporal features. Based on the spatio-temporal features, use a deep learning model to predict the safety area of the robotic arm. Based on the real-time prediction result of the safety area, use the dynamic programming method to plan the optimal motion trajectory of the robotic arm.
2. The dynamic human-machine collaborative safety area prediction method according to claim 1, wherein The step of obtaining the operator's motion data and the robotic arm's motion data in real time respectively specifically includes: Using a vision sensing system to obtain the operator's motion data in real time; the vision sensing system is used to obtain the real-time image of the operator by using a camera; and based on the real-time image, use a human skeleton detection algorithm to determine the operator's motion data. Using a vision sensing system to obtain the robotic arm's motion data in real time. Based on the operator's motion data and the robotic arm's motion data, construct a dynamic dataset for the human-machine collaborative scenario.
3. The dynamic human-machine collaborative safety area prediction method according to claim 1, wherein The step of using a spatio-temporal graph convolutional network to determine spatio-temporal features based on the operator's motion data and the robotic arm's motion data specifically includes: Based on the operator's motion data, use the temporal convolutional layer in the spatio-temporal graph convolutional network to determine the time series of the operator's motion features; and determine the features after temporal convolution based on the time series of the operator's motion features. Determine the adjacency matrix based on the robotic arm's motion data and the operator's motion data. Based on the adjacency matrix and the features after temporal convolution, use the graph convolutional layer in the spatio-temporal graph convolutional network to determine spatio-temporal features.
4. The dynamic human-machine collaborative safety area prediction method according to claim 1, wherein The deep learning model is a long short-term memory network.
5. The dynamic human-machine collaborative safety area prediction method according to claim 1, characterized in that The step of using the dynamic programming method to plan the optimal motion trajectory of the robotic arm based on the real-time prediction result of the safety area specifically includes: Determine the safety area constraint based on the real-time prediction result of the safety area. Determine the constraint conditions based on the safety area constraint. Based on the constraint conditions, use the dynamic programming method to plan the optimal motion trajectory of the robotic arm.
6. The dynamic human-machine collaborative safety area prediction method according to claim 5, wherein The step of determining the safety area constraint based on the real-time prediction result of the safety area specifically includes: Using the formula ||T m -S t ||2≥d safe Determine the safety region constraint; Among them, T m is the position at the end of the robotic arm, S t is the boundary of the safety area at time t, d safe is the safety distance threshold, ||||2 is the L2 norm.
7. The dynamic human-machine collaborative safety area prediction method according to claim 6, wherein The step of determining the constraint conditions based on the safety area constraint specifically includes: Use the formula to determine the constraint conditions; Among them, t0 is the starting time of trajectory planning, t n is the ending time of trajectory planning, T m (t) is the position of the end effector of the robotic arm at time t, || || 2 is the square of the norm.
8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the dynamic human-machine collaborative safety area prediction method according to any one of claims 1-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dynamic human-machine collaborative safety area prediction method according to any one of claims 1-7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the dynamic human-machine collaborative safety area prediction method according to any one of claims 1-7.