Human-machine collaborative safety control method and device based on mixed reality and digital twin
Through the human-machine collaboration safety control method based on mixed reality and digital twins, the problem of human-machine interaction cannot be guaranteed is solved, and the security interaction strategy of human-machine bidirectional collaboration is realized. Through precise measurement and deep reinforcement learning, operational safety is improved Sex and fluency.
Patent Information
- Application Number
- CN202410958101.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-17
AI Technical Summary
The lack of a secure interaction strategy that can achieve bidirectional human-computer collaboration in the prior art has led to the inability to guarantee the security of human-computer interaction.
Using a human-machine collaborative security control method based on mixed reality and digital twins, the human-machine collaborative motion path is determined by obtaining the posture information of physical robots and virtual robots, and performing the posture registration, and using reinforcement learning to plan the path of human-machine collaborative motion, and performing the human-machine Collaborate on safety control actions to ensure operational safety and smoothness.
By accurately measuring the distance and speed of human-machine, using visual assist technology to achieve dynamic visual operation of safe areas, improving operational safety, realizing a safe interaction strategy of human-machine bidirectional collaboration, using virtual robots to perform motion preview and collision detection, and ensuring the movement of physical robots The security and effectiveness of the robot can be driven to independently identify and avoid obstacles through deep reinforcement learning algorithms, ensuring the safety and fluency of human-computer collaboration.
Smart Images

Figure CN118493407B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-machine collaborative intelligent manufacturing and assembly technology, and in particular to a human-machine collaborative safety control method and device based on mixed reality and digital twins. Background Art
[0002] As an important supporting technology and production equipment for modern manufacturing, industrial robots are widely used in mechanical processing, welding, handling and other links. With the development of technology, although pre-programmed robots have high automation efficiency, their configuration time is long and they lack flexibility, making it difficult to adapt to the requirements of customized production. Collaborative robots break through the traditional human-machine isolation working mode and realize human-machine shared workspace, thereby promoting human-machine integration. Before realizing human-machine collaboration, ensuring operator safety is the top priority to avoid accidental collisions. With the increase in the scale and complexity of production and manufacturing, the traditional robot safety protection control strategy based on fixed rules can no longer meet current safety needs.
[0003] At present, researchers have begun to apply augmented reality technology and deep reinforcement learning to human-computer interaction. They mainly use the augmented reality technology of the Internet of Things to superimpose virtual information on the real-world field of vision, enhance people's perception and cognitive ability of the on-site environment, and use deep reinforcement learning algorithms to perform path planning and obstacle avoidance to improve the robot's cognitive decision-making ability. However, there is a defect that the safety of human-computer interaction cannot be guaranteed. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a human-machine collaborative safety control method and device based on mixed reality and digital twins, aiming to solve the problem in the prior art of lack of a safety interaction strategy that enables two-way collaboration between humans and machines.
[0005] In order to achieve the above-mentioned object, the first aspect of the present invention provides a human-machine collaborative safety control method based on mixed reality and digital twins, comprising:
[0006] Acquire first posture information of the physical robot, and acquire second posture information of the virtual robot;
[0007] Based on the first posture information and the second posture information, performing posture registration on the physical robot and the virtual robot to obtain a registration result matrix;
[0008] Based on the registration result matrix and the preset human-machine working area, reinforcement learning is used to perform path planning to determine the human-machine collaborative motion path;
[0009] Based on the human-machine collaborative motion path, a human-machine collaborative safety control action is executed.
[0010] In one implementation, the performing posture registration of the physical robot and the virtual robot based on the first posture information and the second posture information to obtain a registration result matrix includes:
[0011] Calculating a posture registration matrix based on the first posture information and the second posture information;
[0012] Based on the posture registration matrix, the physical robot and the virtual robot are subjected to posture registration and coordinate system registration to obtain a registration result matrix.
[0013] In one embodiment, the method of performing path planning based on the registration result matrix and the preset human-machine working area by using reinforcement learning to determine the human-machine collaborative motion path includes:
[0014] Based on the preset human-machine collaborative control accuracy, several environmental anchor points are constructed;
[0015] Calculating the distance between the physical robot and the virtual robot based on all the environmental anchor points and the human-machine work area;
[0016] Based on the distance, determining a linear velocity at which the physical robot motion is performed;
[0017] Path planning is performed based on the linear speed, deep reinforcement learning principles and inverse kinematics principles to determine the human-machine collaborative motion path.
[0018] In one embodiment, the construction of a plurality of environmental anchor points based on a preset human-machine collaborative control accuracy includes:
[0019] Based on a preset communication frequency, periodically polling the collaborative working status of the physical robot and the virtual robot to determine the current collaborative working status;
[0020] Acquire the current environment anchor point, and determine the current human-machine collaborative control accuracy based on the current collaborative working state and the current environment anchor point;
[0021] If the current human-machine collaborative control accuracy meets the preset human-machine collaborative control accuracy, a number of environmental anchor points are constructed.
[0022] In one embodiment, the path planning based on the linear velocity, deep reinforcement learning principle and inverse kinematics principle to determine the human-machine collaborative motion path includes:
[0023] Calculating the joint angular velocity based on the linear velocity, deep reinforcement learning principles and inverse kinematics principles, and generating the motion trajectory of the physical robot using the joint angular velocity and a preset target posture;
[0024] Based on the motion trajectory of the physical robot, calculating the joint space combination sequence of the physical robot;
[0025] Mapping the joint space combination sequence to the joint space of the virtual robot to determine the joint space combination sequence of the virtual robot;
[0026] Determining a motion path of the virtual robot based on a joint space combination sequence of the virtual robot;
[0027] Based on the motion trajectory of the physical robot and the motion path of the virtual robot, a human-machine collaborative motion path is determined.
[0028] In one embodiment, determining the motion path of the virtual robot based on the joint space combination sequence of the virtual robot includes:
[0029] Calculating the joint motion values of the virtual robot based on the joint space combination sequence of the virtual robot;
[0030] Based on the joint motion values and the human-machine working area, a motion path of the virtual robot is determined.
[0031] In one embodiment, the performing of a human-machine collaborative safety control action based on the human-machine collaborative motion path includes:
[0032] If the human-machine collaborative motion path does not meet at least one of the preset motion safety standards or the human-machine collaborative control accuracy, several new environmental anchor points are reconstructed, and the path planning process is executed based on the new environmental anchor points until the human-machine collaborative motion path meets the preset motion safety standards and the human-machine collaborative control accuracy, and the human-machine collaborative safety control action is executed.
[0033] A second aspect of the present invention provides a human-machine collaborative safety control device based on mixed reality and digital twins, the system comprising:
[0034] An information acquisition module, used to acquire first posture information of the physical robot and second posture information of the virtual robot;
[0035] A registration module, configured to perform posture registration on the physical robot and the virtual robot based on the first posture information and the second posture information to obtain a registration result matrix;
[0036] A human-machine collaborative path planning module is used to perform path planning based on the registration result matrix and the preset human-machine working area using reinforcement learning to determine the human-machine collaborative motion path;
[0037] A control execution module is used to execute human-machine collaborative safety control actions based on the human-machine collaborative motion path.
[0038] The third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and a human-machine collaborative safety control program based on mixed reality and digital twins stored in the memory and executable on the processor. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by the processor, any step of the above-mentioned human-machine collaborative safety control method based on mixed reality and digital twins is implemented.
[0039] A fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a human-machine collaborative safety control program based on mixed reality and digital twins. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by a processor, the steps of any one of the above-mentioned human-machine collaborative safety control methods based on mixed reality and digital twins are implemented.
[0040] Compared with the prior art, the beneficial effects of this solution are as follows:
[0041] The present invention achieves dynamic visualization operation in the safe area by accurately measuring the human-machine distance and speed and utilizing visual assistance technology to improve operational safety. It also maps the joint space combination sequence of the physical robot to the action space of the virtual robot through the virtual-real space mapping method, thereby realizing a safe interaction strategy for two-way collaboration between man and machine. The virtual robot is used to realize motion preview and collision detection of actual operations, thereby ensuring the safety and effectiveness of the physical robot's actions. The deep reinforcement learning algorithm is used to drive the robot to autonomously identify and actively avoid potential obstacles, thereby ensuring the safety and smoothness of human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0043] Figure 1 This is a brief flow chart of the human-machine collaborative safety control method based on mixed reality and digital twins of the present invention;
[0044] Figure 2 It is a detailed flow chart of the human-machine collaborative safety control method based on mixed reality and digital twins of the present invention;
[0045] Figure 3It is a module schematic diagram of the human-machine collaborative safety control device based on mixed reality and digital twins of the present invention;
[0046] Figure 4 It is a schematic diagram of the structure of the intelligent terminal of the present invention. DETAILED DESCRIPTION
[0047] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.
[0048] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0049] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0050] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0051] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0053] In the face of the existing problem of lack of safe interaction strategies that can enable two-way collaboration between humans and machines, the present invention proposes a human-machine collaborative safety control method based on mixed reality and digital twins. It mainly relies on the perception and control functions of the mixed reality device platform, and through real-time perception of the human-machine collaborative scene, a full-stage human-machine mutual cognition safety control model is constructed, which mainly includes the following three aspects: First, by accurately measuring the human-machine distance and speed, and using visual assistance technology to realize dynamic visualization operation in the safe area, thereby improving operational safety; second, through the method of virtual-real space mapping, the joint space combination sequence of the physical robot is mapped to the action space where the virtual robot is located, and the virtual robot is used to realize motion preview and collision detection of actual operations, thereby ensuring the safety and effectiveness of the physical robot's actions; third, combining the above two safety interaction strategies, the robot is driven by a deep reinforcement learning algorithm to autonomously identify and actively avoid potential obstacles, thereby ensuring the safety and smoothness of human-machine collaboration.
[0054] The embodiment of the present invention provides a human-machine collaborative safety control method based on mixed reality and digital twins, which is deployed on electronic devices such as computers and servers. The application scenario is the situation where a physical robot and the virtual robot perform human-machine collaborative safety control. The types of the above-mentioned physical robots and virtual robots are not limited. The physical robot can be an industrial robot, etc., and the virtual robot can be driven and generated by a mixed reality wearable device, etc. Specifically, Figure 1 and Figure 2 As shown, the steps of the method in this embodiment include:
[0055] Step S100: obtaining first posture information of the physical robot and obtaining second posture information of the virtual robot;
[0056] Specifically, the industrial robot arm controller of the physical robot is started and initialized to the ready-to-work state, and the initialization posture information of the physical robot, i.e., the first posture information, is provided for the subsequent human-machine posture alignment. Specifically, it includes but is not limited to starting the robot arm control system of the physical robot, loading the initialization parameters of the robot arm control system (such as speed, acceleration, collision force threshold and load, etc.), and starting the preparation position through the sensor calibration initial program to ensure that the robot arm is in the ready-to-work state.
[0057] The virtual robot is generated by driving the mixed reality wearable device, and the digital twin of the virtual robot based on the real-time physical robot is established, providing the initialization posture information of the virtual robot, i.e. the second posture information, for the subsequent human-machine posture alignment. The specific operations include but are not limited to the startup of the virtual robot, the initialization of the virtual robot, and the preliminary synchronization and alignment with the physical robot, which lays a good foundation for the subsequent posture alignment of the physical robot and the virtual robot.
[0058] Step S200: Based on the first posture information and the second posture information, the physical robot and the virtual robot are aligned to obtain an alignment result matrix;
[0059] Specifically, a posture registration matrix is calculated based on the first posture information and the second posture information; and based on the posture registration matrix, the physical robot and the virtual robot are posture-registered and coordinate-registered to obtain a registration result matrix.
[0060] According to the posture registration matrix calculated multiple times in the robot posture registration program, the mixed reality device is used to drive the 6D posture registration of the physical robot and the virtual robot in the digital twin environment. Specifically, it includes but is not limited to the collection of data related to the postures of the physical robot and the virtual robot, multiple registration calculations, and real-time debugging and correction based on the mixed reality device.
[0061] The mixed reality device is used to maintain a fixed communication frequency between the physical robot and the virtual robot, and the status of the physical and virtual robots is polled regularly to determine whether the communication real-time and status meet the accuracy requirements of the robot motion execution. If not, step S100 is repeated and the robot posture alignment process is performed again. Specifically, it includes but is not limited to setting the communication frequency, status polling, data analysis and decision feedback.
[0062] Among them, the robot posture registration procedure specifically includes: setting the industrial robot arm controller parameters and safety parameters of the physical robot in the collaborative robot controller, and setting the initial physical robot posture, including but not limited to inputting initial parameters and calibration posture; starting the mixed reality device to create a virtual robot digital twin and projecting it into the workspace to overlap with the physical robot's perspective; initializing the physical robot controller motor, changing the physical robot posture and reading the physical robot's initialization posture information, including but not limited to motor start-up, posture adjustment and data reading; according to the physical robot posture information and the virtual robot posture information, taking multiple sets of data to calculate the displacement rotation matrix and combining the least squares method and eigenvalue regression to perform posture matrix calculation and registration to obtain the registration result matrix.
[0063] Step S300: Based on the registration result matrix and the preset human-machine working area, reinforcement learning is used to perform path planning to determine the human-machine collaborative motion path;
[0064] Specifically, based on the preset human-machine collaborative control accuracy, a number of environmental anchor points are constructed; based on all the environmental anchor points and the human-machine working area, the distance between the physical robot and the virtual robot is calculated; based on the distance, the linear speed of the physical robot's movement is determined; path planning is performed based on the linear speed, deep reinforcement learning principles and inverse kinematics principles to determine the human-machine collaborative motion path.
[0065] Among them, based on the preset human-machine collaborative control accuracy, a number of environmental anchor points are constructed, specifically including: based on the preset communication frequency, the collaborative working status of the physical robot and the virtual robot is polled regularly to determine the current collaborative working status; the current environmental anchor point is obtained, and the current human-machine collaborative control accuracy is determined based on the current collaborative working status and the current environmental anchor point; if the current human-machine collaborative control accuracy meets the human-machine collaborative control accuracy, a number of environmental anchor points are constructed.
[0066] If the collaborative state of the physical manipulator and the virtual robot meets the control accuracy requirements, the virtual robot is started to reconstruct the environmental anchor points, and based on all the environmental anchor points and the human-machine working area, the distance between the physical robot and the virtual robot, that is, the distance between the human and the machine, is calculated, where the "human" in human-machine refers to the virtual robot, and the "machine" in human-machine refers to the physical robot; based on the distance, the linear velocity of the end effector executed by the robot motion is determined, and the joint angular velocity is calculated through the Jacobian matrix, and this is used as the action space setting for deep reinforcement learning to support high-precision and safe speed human-machine collaborative operations. Specifically, it includes but is not limited to environmental scanning, anchor point recognition, human-machine distance calculation, and end effector speed setting.
[0067] After the end effector motion speed is set, a collaborative feedback control based on human-machine two-way communication under a mixed reality device is constructed, specifically including: calculating the joint angular velocity based on the linear velocity, deep reinforcement learning principles and inverse kinematics principles, and using the joint angular velocity and a preset target posture to generate the motion trajectory of the physical robot; calculating the joint space combination sequence of the physical robot based on the motion trajectory of the physical robot; mapping the joint space combination sequence to the joint space of the virtual robot to determine the joint space combination sequence of the virtual robot; based on the joint space combination sequence of the virtual robot, solving the joint motion value of the virtual robot based on the joint space combination sequence of the virtual robot; determining the motion path of the virtual robot based on the joint motion value and the human-machine working area; determining the human-machine collaborative motion path based on the motion trajectory of the physical robot and the motion path of the virtual robot.
[0068] In the effective human-machine working area, real-time or planned actions issued by gestures, buttons, voice, etc. are combined with the obtained multimodal input information, and the operation trajectory of the end effector is generated using a deep reinforcement learning algorithm, and the joint space combination sequence required by the physical robot is calculated by an inverse kinematics solver. The process of calculating the joint space combination sequence required by the physical robot through an inverse kinematics solver includes: establishing a robot model, defining the target position and posture sequence, using the inverse kinematics principle to calculate the joint angle corresponding to each target point, and generating a joint space combination sequence to ensure that the robot can accurately execute the task path, including but not limited to multimodal input parsing, deep reinforcement learning algorithm principles, and inverse kinematics principle solving. The joint space combination sequence is then mapped to the joint space of the virtual robot to achieve path planning for the follow-up operation.
[0069] In a preferred embodiment, in order to solve the problem of robot motion generation in complex industrial tasks, the motion planning method based on deep reinforcement learning includes the following steps: first, according to the operator's job requirements, the target 6D posture or motion trajectory of the virtual robot is set to ensure the accuracy and executability of the task target; then, according to the current posture and target posture of the virtual robot, the running speed is determined and the neural network controller generated by deep reinforcement learning is combined with the inverse kinematics solver to calculate the joint value of the virtual robot at the target posture, thereby achieving efficient and accurate posture control. Then, the spatial computing power and visualization ability of mixed reality are used to check whether there is a potential collision possibility during the operation process. Then, the posture conversion matrix and calibration data of the virtual robot are combined to calculate the posture of the physical robot to ensure the precise alignment between the virtual and physical robots. In the motion planning based on deep reinforcement learning, the input of the deep reinforcement learning motion planning module is the target posture of the virtual robot in the mixed reality environment, and the output is the joint state of the robot at the target posture calculated by the inverse kinematics solver. Through the posture conversion configuration module, these joint states are output as the joint states of the physical robot, including the base joint, shoulder joint, elbow joint, wrist 1 joint, wrist 2 joint and wrist 3 joint. Finally, these joint states are output to the physical robot controller to ensure that the robot can complete the task according to the predetermined trajectory and posture, achieving efficient task execution and accurate action generation. Through the above steps, the present invention provides a robot safe motion planning method combined with deep reinforcement learning, which can effectively deal with the motion generation problem in complex industrial tasks and improve the flexibility and accuracy of robot operation.
[0070] Step S400: Execute a human-machine collaborative safety control action based on the human-machine collaborative motion path.
[0071] Specifically, if the human-machine collaborative motion path does not meet at least one of the preset motion safety standards or the human-machine collaborative control accuracy, several new environmental anchor points are reconstructed, and the path planning process is executed based on the new environmental anchor points until the human-machine collaborative motion path meets the preset motion safety standards and the human-machine collaborative control accuracy, and the human-machine collaborative safety control action is executed.
[0072] Based on the virtual robot posture and the specific rotation angle and speed values of each joint calculated by inverse kinematics, combined with the distance space calculation of the mixed reality device, the robot's Cartesian coordinates can ensure that the trajectory generated by the robot does not exceed the effective working range. If it does not exceed, the physical robot joint space transformation can be triggered. The operation includes safe distance calculation, trajectory verification and joint space transformation.
[0073] To determine whether the human-machine collaborative motion path meets the robot's motion safety requirements and execution accuracy, the mixed reality device calculates and reconstructs the environment and checks whether there is a collision between the robot's digital twin and the reconstructed environment. If not, it meets the collision-free safety specification, supplemented by human subjective inspection. If not, the human-machine collaborative motion path planning is re-performed. If the planned human-machine collaborative motion path meets the robot's execution accuracy in the spatial coordinate system within the safety range and there is no potential risk of collision, the solved joint commands can drive the physical robot to perform motion execution in the workshop, completing the human-machine two-way safe collaborative feedback control.
[0074] In summary, the beneficial effects of the present invention include:
[0075] The present invention can capture the multimodal input (such as gestures, voice, touch, etc.) of human operators, the distance between humans and machines, and scene information in real time, thereby providing instant and accurate feedback for human-machine collaborative scenes. The virtual robot in the mixed reality device provides the robot controller with multimodal control input (including tactile, visual and voice feedback), which not only improves the intuitiveness and convenience of operation, but also enhances the efficiency and accuracy of complex human-machine collaborative work. The physical robot and the virtual robot achieve seamless real-time data transmission through the digital twin communication mechanism, ensuring the real-time operation of the robot. At the same time, through the posture registration matrix, the physical robot can accurately reproduce the pre-planned trajectory, ensuring the high precision and consistency of the operation.
[0076] The method of the present invention is highly adaptable and versatile and can adapt to a variety of manufacturing scenarios and tasks. Whether it is different characters, complex scenarios or diverse task requirements, the system can provide effective security protection. Its modular design allows the system to be flexibly configured and expanded in different application scenarios, thereby meeting the needs of human-machine collaboration in different industrial environments. Through intelligent perception and analysis of tasks and environments, the system can dynamically adjust its working mode and security strategy to adapt to real-time changing operating environments and task requirements.
[0077] The present invention decomposes safety protection measures into progressive, multi-level strategies through module decoupling. This design can not only avoid the problems of insufficient protection and reduced accuracy that may be caused by a single module during deployment, but also improve the overall safety and reliability of the system. Reinforcement learning motion planning based on inverse kinematics enables the virtual robot to intuitively accept and understand the flexible operating posture of the robot controller. Through inverse calculation, the system can accurately solve the joint values of the robot in the target posture and pass it to the physical robot to ensure that it can perform precise and flexible task operations. In addition, the system also integrates multi-sensor data fusion technology to improve the accuracy of environmental perception and task execution, thereby further improving the safety and efficiency of robot operations.
[0078] like Figure 3 As shown, corresponding to the above-mentioned human-machine collaborative safety control method based on mixed reality and digital twins, an embodiment of the present invention further provides a human-machine collaborative safety control device based on mixed reality and digital twins, and the above-mentioned human-machine collaborative safety control device based on mixed reality and digital twins includes:
[0079] An information acquisition module 310 is used to acquire first posture information of the physical robot and to acquire second posture information of the virtual robot;
[0080] A registration module 320, configured to perform posture registration on the physical robot and the virtual robot based on the first posture information and the second posture information to obtain a registration result matrix;
[0081] A human-machine collaborative path planning module 330 is used to perform path planning using reinforcement learning based on the registration result matrix and the preset human-machine working area to determine the human-machine collaborative motion path;
[0082] The control execution module 340 is used to execute the human-machine collaborative safety control action based on the human-machine collaborative motion path.
[0083] Specifically, in this embodiment, the specific functions of the above-mentioned human-machine collaborative safety control device based on mixed reality and digital twins can also refer to the corresponding description in the above-mentioned human-machine collaborative safety control method based on mixed reality and digital twins, and will not be repeated here.
[0084] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 4 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected via a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a human-machine collaborative safety control program based on mixed reality and digital twins. The internal memory provides an environment for the operation of the operating system and the human-machine collaborative safety control program based on mixed reality and digital twins in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by the processor, any step of the human-machine collaborative safety control method based on mixed reality and digital twins is implemented. The display screen of the intelligent terminal may be a liquid crystal display screen or an electronic ink display screen.
[0085] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0086] In one embodiment, a smart terminal is provided, which includes a memory, a processor, and a human-machine collaborative safety control program based on mixed reality and digital twins stored in the memory and executable on the processor. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by the processor, the steps of any one of the human-machine collaborative safety control methods based on mixed reality and digital twins provided in an embodiment of the present invention are implemented.
[0087] An embodiment of the present invention also provides a computer-readable storage medium, on which is stored a human-machine collaborative safety control program based on mixed reality and digital twins. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by a processor, the steps of any one of the human-machine collaborative safety control methods based on mixed reality and digital twins provided in an embodiment of the present invention are implemented.
[0088] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0089] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0090] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0091] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0092] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0093] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A human-machine collaborative safety control method based on mixed reality and digital twins, characterized in that: The following steps are involved: Acquire first posture information of the physical robot, and acquire second posture information of the virtual robot; Based on the first posture information and the second posture information, performing posture registration on the physical robot and the virtual robot to obtain a registration result matrix; Based on the registration result matrix and the preset human-machine working area, reinforcement learning is used to perform path planning to determine the human-machine collaborative motion path; Based on the human-machine collaborative motion path, executing human-machine collaborative safety control actions; The method of performing path planning based on the registration result matrix and the preset human-machine working area by using reinforcement learning to determine the human-machine collaborative motion path includes: Based on the preset human-machine collaborative control accuracy, several environmental anchor points are constructed; Based on all the environmental anchor points and the human-machine work area, the distance between the physical robot and the virtual robot is calculated; based on the distance, the linear speed of the physical robot's motion execution is determined; based on the linear speed, deep reinforcement learning principles and inverse kinematics principles, the joint angular velocity is calculated, and the joint angular velocity and the preset target posture are used to generate the motion trajectory of the physical robot; based on the motion trajectory of the physical robot, the joint space combination sequence of the physical robot is calculated; the joint space combination sequence is mapped to the joint space of the virtual robot to determine the joint space combination sequence of the virtual robot; based on the joint space combination sequence of the virtual robot, the joint motion value of the virtual robot is solved; based on the joint motion value and the human-machine work area, the motion path of the virtual robot is determined; based on the motion trajectory of the physical robot and the motion path of the virtual robot, the human-machine collaborative motion path is determined; Within the effective human-machine working area, real-time or planned actions issued by gestures, buttons or voice are combined with the obtained multimodal input information to generate the operation trajectory of the end effector of the physical robot using a deep reinforcement learning algorithm, and the joint space combination sequence required by the physical robot is calculated through an inverse kinematics solver.
2. The human-machine collaborative safety control method based on mixed reality and digital twins according to claim 1 is characterized in that: The step of performing posture registration on the physical robot and the virtual robot based on the first posture information and the second posture information to obtain a registration result matrix includes: Calculating a posture registration matrix based on the first posture information and the second posture information; Based on the posture registration matrix, the physical robot and the virtual robot are subjected to posture registration and coordinate system registration to obtain a registration result matrix.
3. The human-machine collaborative safety control method based on mixed reality and digital twins according to claim 1 is characterized in that: Based on the preset human-machine collaborative control accuracy, several environmental anchor points are constructed, including: Based on a preset communication frequency, periodically polling the collaborative working status of the physical robot and the virtual robot to determine the current collaborative working status; Acquire the current environment anchor point, and determine the current human-machine collaborative control accuracy based on the current collaborative working state and the current environment anchor point; If the current human-machine collaborative control accuracy meets the preset human-machine collaborative control accuracy, a number of environmental anchor points are constructed.
4. The human-machine collaborative safety control method based on mixed reality and digital twins according to claim 1 is characterized in that: The performing of a human-machine collaborative safety control action based on the human-machine collaborative motion path includes: If the human-machine collaborative motion path does not meet at least one of the preset motion safety standards or the human-machine collaborative control accuracy, several new environmental anchor points are reconstructed, and the path planning process is executed based on the new environmental anchor points until the human-machine collaborative motion path meets the preset motion safety standards and the human-machine collaborative control accuracy, and the human-machine collaborative safety control action is executed.
5. A human-machine collaborative safety control device based on mixed reality and digital twins, characterized in that: Applied to implement the human-machine collaborative safety control method based on mixed reality and digital twins as described in any one of claims 1 to 4, the device comprises: An information acquisition module, used to acquire first posture information of the physical robot and to acquire second posture information of the virtual robot; A registration module, configured to perform posture registration on the physical robot and the virtual robot based on the first posture information and the second posture information to obtain a registration result matrix; A human-machine collaborative path planning module is used to perform path planning based on the registration result matrix and the preset human-machine working area using reinforcement learning to determine the human-machine collaborative motion path; A control execution module is used to execute human-machine collaborative safety control actions based on the human-machine collaborative motion path.
6. Intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a human-machine collaborative safety control program based on mixed reality and digital twins stored in the memory and executable on the processor. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by the processor, the steps of the human-machine collaborative safety control method based on mixed reality and digital twins are implemented as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a human-machine collaborative safety control program based on mixed reality and digital twins. When the human-machine collaborative safety control program based on mixed reality and digital twins is executed by the processor, the steps of the human-machine collaborative safety control method based on mixed reality and digital twins are implemented as described in any one of claims 1-4.
