Virtual-robot and physical-robot collaborative safety control method and apparatus based on mixed reality and digital twin

Abstract

Provided in the present invention are a virtual-robot and physical-robot collaborative safety control method and apparatus based on mixed reality and a digital twin, which method and apparatus specifically relate to the technical field of smart manufacturing and assembly involving virtual-robot and physical-robot collaboration. In the solution, the method comprises: acquiring first posture information of a physical robot, and acquiring second posture information of a virtual robot; on the basis of the first posture information and the second posture information, performing pose registration on the physical robot and the virtual robot, so as to obtain a registration result; on the basis of the registration result and a preset virtual-robot and physical-robot workspace in combination with deep reinforcement learning, performing path planning to determine a virtual-robot and physical-robot collaborative motion path; and on the basis of the virtual-robot and physical-robot collaborative motion path, executing a virtual-robot and physical-robot collaborative safety control action. The solution uses a mixed-reality device to extract virtual-robot and physical-robot posture information, implements virtual-robot and physical-robot pose registration by means of a virtual-real space mapping method, and designs a bidirectional virtual-robot and physical-robot collaborative safety interaction policy based on deep reinforcement learning, thereby effectively improving the safety and effectiveness of a virtual-robot and physical-robot interaction operation.

Description

Man-machine cooperation safety control method and device based on mixed reality and digital twin TECHNICAL FIELD

[0001] The present application relates to the field of man-machine collaborative intelligent manufacturing assembly technology, in particular to a man-machine cooperation safety control method and device based on mixed reality and digital twin. BACKGROUND

[0002] Industrial robots, as an important supporting technology and production equipment of modern manufacturing, are widely used in machining, welding and handling. With the development of technology, although the pre-programmed robot has high automation efficiency, it has long configuration time and lacks flexibility, which is difficult to meet the requirements of customized production. Collaborative robots break through the traditional man-machine isolation mode and realize man-machine sharing of working space, thereby promoting man-machine integration. Before realizing man-machine cooperation, ensuring the safety of operators is the primary task to avoid accidental collision. With the increase of production scale and complexity, the traditional robot safety protection control strategy based on fixed rules cannot meet the current safety needs.

[0003] At present, researchers begin to apply augmented reality technology and deep reinforcement learning to human-computer interaction, mainly using the augmented reality technology of Internet of Things to superimpose virtual information in the field of view of the real world, enhance the perception and cognition ability of people to the scene environment, and through the deep reinforcement learning algorithm for path planning and obstacle avoidance, to improve the cognitive decision-making ability of the robot, but there is a defect that the safety of human-computer interaction cannot be guaranteed. SUMMARY

[0004] In view of the above shortcomings of the prior art, the present application aims to provide a man-machine cooperation safety control method and device based on mixed reality and digital twin, which aims to solve the problem of lack of safe interaction strategy for man-machine bidirectional cooperation in the prior art.

[0005] In order to achieve the above purpose, the first aspect of the present application provides a man-machine cooperation safety control method based on mixed reality and digital twin, comprising:

[0006] Obtaining first attitude information of a physical robot, and obtaining second attitude information of a virtual robot;

[0007] Based on the first attitude information and the second attitude information, the physical robot and the virtual robot are pose registered to obtain a registration result matrix;

[0008] Based on the registration result matrix and a preset man-machine working area, path planning is performed by using reinforcement learning to determine a man-machine cooperative motion path;

[0009] Perform a human-robot collaborative safety control action based on the human-robot collaborative motion path.

[0010] In an embodiment, the physical robot and the virtual robot are pose-registered based on the first pose information and the second pose information to obtain a registration result matrix, including:

[0011] A pose registration matrix is calculated based on the first pose information and the second pose information.

[0012] The physical robot and the virtual robot are pose-registered and coordinate system-registered based on the pose registration matrix to obtain a registration result matrix.

[0013] In an embodiment, the human-robot collaborative motion path is determined by path planning using reinforcement learning based on the registration result matrix and a preset human-robot work area, including:

[0014] A plurality of environmental anchor points are constructed based on a preset human-robot collaborative control accuracy.

[0015] A distance between the physical robot and the virtual robot is calculated based on all the environmental anchor points and the human-robot work area.

[0016] A linear velocity of motion performed by the physical robot is determined based on the distance.

[0017] The human-robot collaborative motion path is determined by path planning based on the linear velocity, a deep reinforcement learning principle, and an inverse kinematics principle.

[0018] In an embodiment, the plurality of environmental anchor points are constructed based on a preset human-robot collaborative control accuracy, including:

[0019] A current collaborative work state of the physical robot and the virtual robot is polled at a preset communication frequency to determine a current collaborative work state.

[0020] A current human-robot collaborative control accuracy is determined based on the current collaborative work state and a current environmental anchor point.

[0021] If the current human-robot collaborative control accuracy meets the preset human-robot collaborative control accuracy, the plurality of environmental anchor points are constructed.

[0022] In an embodiment, the human-robot collaborative motion path is determined by path planning based on the linear velocity, a deep reinforcement learning principle, and an inverse kinematics principle, including:

[0023] Calculate joint angular velocity based on the linear velocity, deep reinforcement learning principle and inverse kinematics principle, and generate the motion trajectory of the physical robot by using the joint angular velocity and the preset target pose;

[0024] Calculate the joint space combination sequence of the physical robot based on the motion trajectory of the physical robot;

[0025] Map the joint space combination sequence to the joint space of the virtual robot to determine the joint space combination sequence of the virtual robot;

[0026] Determine the motion path of the virtual robot based on the joint space combination sequence of the virtual robot;

[0027] Determine the human-robot collaborative motion path based on the motion trajectory of the physical robot and the motion path of the virtual robot.

[0028] In an embodiment, determining the motion path of the virtual robot based on the joint space combination sequence of the virtual robot comprises:

[0029] Solving the joint motion value of the virtual robot based on the joint space combination sequence of the virtual robot;

[0030] Determining the motion path of the virtual robot based on the joint motion value and the human-robot work area.

[0031] In an embodiment, performing human-robot collaborative safety control action based on the human-robot collaborative motion path comprises:

[0032] If the human-robot collaborative motion path does not meet at least one of the preset motion safety standard or the human-robot collaborative control accuracy, a plurality of new environmental anchor points are reconstructed, and the path planning process is performed based on the new environmental anchor points until the human-robot collaborative motion path meets the preset motion safety standard and the human-robot collaborative control accuracy, and the human-robot collaborative safety control action is performed.

[0033] The second aspect of the application provides a human-robot collaborative safety control device based on mixed reality and digital twin, the system comprises:

[0034] An information acquisition module is configured to acquire first pose information of a physical robot and second pose information of a virtual robot;

[0035] A registration module is configured to perform pose registration on the physical robot and the virtual robot based on the first pose information and the second pose information to obtain a registration result matrix;

[0036] The man-machine collaborative path planning module is configured to perform path planning by using reinforcement learning based on the registration result matrix and a preset man-machine work area, and determine a man-machine collaborative motion path.

[0037] The control execution module is configured to perform a man-machine collaborative safety control action based on the man-machine collaborative motion path.

[0038] The third aspect of the present application provides an intelligent terminal, which comprises a memory, a processor, and a man-machine collaborative safety control program based on mixed reality and digital twin stored on the memory and executable on the processor.

[0039] The fourth aspect of the present application provides a computer readable storage medium, which stores a man-machine collaborative safety control program based on mixed reality and digital twin, and the man-machine collaborative safety control program based on mixed reality and digital twin implements the steps of any one of the above-mentioned man-machine collaborative safety control methods based on mixed reality and digital twin when executed by a processor.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] The present application can improve the operation safety by accurately measuring the distance and speed between the man and the machine, and using visual auxiliary technology to realize dynamic visualization operation of the safety area. The present application can realize the safety interaction strategy of man-machine bidirectional collaboration by mapping the joint space combination sequence of the physical robot to the action space of the virtual robot through the method of virtual-real space mapping, and using the virtual robot to realize the motion preview and collision detection of the actual operation, so as to ensure the safety and effectiveness of the physical robot action. The robot can autonomously identify and actively avoid potential obstacles by using the deep reinforcement learning algorithm, so as to ensure the safety and fluency of the man-machine collaboration. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0043] Fig. 1 is a brief flowchart of the man-machine collaborative safety control method based on mixed reality and digital twin of the present application;

[0044] Fig. 2 is a detailed flowchart of the man-machine collaborative safety control method based on mixed reality and digital twin of the present application;

[0045] Fig. 3 is a schematic diagram of modules of the mixed reality and digital twin based human-machine collaboration safety control device of the present application;

[0046] Fig. 4 is a schematic diagram of the structure of the intelligent terminal of the present application. DETAILED DESCRIPTION

[0047] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0048] It is to be understood that the terminology "including", when used in the present specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] It is also to be understood that the terminology used in the present specification and the appended claims is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0050] It will be further understood that the terms "comprises" and / or "comprising", when used in this specification and the following claims, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0051] The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0053] The present application faces the problem of the lack of safe interaction strategy capable of bidirectional human-machine cooperation, and proposes a human-machine cooperation safety control method based on mixed reality and digital twin. The method mainly uses the perception and control functions of the mixed reality device platform to build a full-stage human-machine mutual cognitive safety control model by real-time perception of human-machine cooperation scenes. The model mainly includes the following three aspects: 1. Precise measurement of human-machine distance and speed is used to realize dynamic visualization of the safety area through visual auxiliary technology, thereby improving operation safety; 2. The joint space combination sequence of the physical robot is mapped to the action space of the virtual robot through virtual-real space mapping, and the virtual robot is used to realize motion preview and collision detection of the actual operation, thereby ensuring the safety and effectiveness of the physical robot action; 3. The safety interaction strategies of the above two aspects are combined, and a deep reinforcement learning algorithm is used to drive the robot to autonomously identify and actively avoid potential obstacles, thereby ensuring the safety and smoothness of human-machine cooperation.

[0054] The embodiment of the present application provides a human-machine cooperation safety control method based on mixed reality and digital twin, which is deployed on electronic devices such as computers and servers, and is applied to the case of human-machine cooperation safety control of physical robots and virtual robots. The types of the above-mentioned physical robots and virtual robots are not limited, and the physical robots can be industrial robots, etc., and the virtual robots can be generated by mixed reality wearable devices, etc. Specifically, as shown in FIG. 1 and FIG. 2, the steps of the method of the present embodiment include:

[0055] Step S100: acquiring first pose information of a physical robot, and acquiring second pose information of a virtual robot;

[0056] Specifically, the industrial robot arm controller of the physical robot is started, and is initialized to a standby state, thereby providing physical robot initialization pose information, i.e. first pose information, for subsequent human-machine pose registration. Specifically, but not limited to, the mechanical arm control system of the physical robot is started, the initialization parameters (such as speed, acceleration, collision force threshold and load, etc.) of the mechanical arm control system are loaded, and the initial program is calibrated through the sensor to start the preparation position, so as to ensure that the mechanical arm is in a standby state.

[0057] The virtual robot is generated by a mixed reality wearable device, and a virtual robot digital twin based on a real-time physical robot is established, thereby providing virtual robot initialization pose information, i.e. second pose information, for subsequent human-machine pose registration. The specific operation includes, but is not limited to, starting of the virtual robot, initialization of the virtual robot, and preliminary synchronization and alignment with the physical robot, thereby laying a good foundation for subsequent pose registration of the physical robot and the virtual robot.

[0058] Step S200: based on the first attitude information and the second attitude information, performing pose registration on the physical robot and the virtual robot to obtain a registration result matrix;

[0059] Specifically, based on the first attitude information and the second attitude information, a pose registration matrix is calculated; based on the pose registration matrix, the physical robot and the virtual robot are registered in pose and coordinate system to obtain a registration result matrix.

[0060] According to the pose registration matrix calculated by the robot pose registration program in multiple registration, the physical robot and the virtual robot are driven by the mixed reality device to perform 6D pose registration in the digital twin environment. Specifically, it includes but is not limited to the collection of data related to the pose of the physical robot and the virtual robot, multiple registration calculation, and real-time debugging and correction based on the mixed reality device, etc.

[0061] The physical robot and the virtual robot are kept in fixed communication frequency by the mixed reality device, and the state coordination of the physical and virtual robots is polled at regular intervals to determine whether the communication real-time and state meet the accuracy requirements of robot motion execution. If not, repeat step S100 and perform the robot pose registration process again. Specifically, it includes but is not limited to setting the communication frequency, state polling, data analysis and decision feedback.

[0062] The robot pose registration program specifically includes: setting the industrial robot controller parameters and safety parameters of the physical robot in the collaborative robot controller, and setting the initial physical robot pose, specifically including but not limited to inputting initial parameters and calibrating the pose; starting the mixed reality device to create a virtual robot digital twin and projecting it into the workspace to coincide with the physical robot's perspective; initializing the physical robot controller motor, transforming the physical robot pose and reading the initial attitude information of the physical robot, specifically including but not limited to starting the motor, adjusting the pose and reading the data; according to the physical robot attitude information and the virtual robot attitude information, multiple groups of data are taken to calculate the displacement rotation matrix, and the least square method and eigenvalue regression are used to calculate and register the attitude matrix to obtain the registration result matrix.

[0063] Step S300: based on the registration result matrix and the preset human-machine working area, path planning is performed by reinforcement learning to determine the human-robot collaborative motion path;

[0064] Specifically, based on a preset human-robot collaborative control accuracy, a plurality of environment anchor points are constructed; based on all the environment anchor points and the human-robot working area, the distance between the physical robot and the virtual robot is calculated; based on the distance, the linear velocity of the physical robot motion execution is determined; based on the linear velocity, the principle of deep reinforcement learning and the principle of inverse kinematics, the path planning is carried out to determine the human-robot collaborative motion path.

[0065] Wherein, based on the preset human-robot collaborative control accuracy, a plurality of environment anchor points are constructed, specifically including: based on the preset communication frequency, the collaborative working state of the physical robot and the virtual robot is polled regularly to determine the current collaborative working state; the current environment anchor point is acquired, and based on the current collaborative working state and the current environment anchor point, the current human-robot collaborative control accuracy is determined; if the current human-robot collaborative control accuracy meets the human-robot collaborative control accuracy, a plurality of environment anchor points are constructed.

[0066] If the collaborative state of the physical robot and the virtual robot meets the control accuracy requirement, the virtual robot is started to reconstruct the environment anchor point, and based on all the environment anchor points and the human-robot working area, the distance between the physical robot and the virtual robot, i.e. the distance between the human and the robot, is calculated, wherein the "human" in the human-robot represents the virtual robot, and the "robot" in the human-robot represents the physical robot; based on the distance, the end effector linear velocity of the robot motion execution is determined and the joint angular velocity is calculated through Jacobian matrix, and this is used as the action space setting of deep reinforcement learning to support high-precision and safe-speed human-robot collaborative work. Specifically, but not limited to, environment scanning, anchor point identification, human-robot distance calculation and end effector velocity setting.

[0067] After setting the end effector motion velocity, the collaborative feedback control based on the human-robot bidirectional communication under the mixed reality device is constructed, specifically including: based on the linear velocity, the principle of deep reinforcement learning and the principle of inverse kinematics, the joint angular velocity is calculated, and the joint angular velocity and the preset target posture are used to generate the motion trajectory of the physical robot; based on the motion trajectory of the physical robot, the joint space combination sequence of the physical robot is calculated; the joint space combination sequence is mapped to the joint space of the virtual robot to determine the joint space combination sequence of the virtual robot; based on the joint space combination sequence of the virtual robot, the joint motion value of the virtual robot is calculated based on the joint space combination sequence of the virtual robot; based on the joint motion value and the human-robot working area, the motion path of the virtual robot is determined; based on the motion trajectory of the physical robot and the motion path of the virtual robot, the human-robot collaborative motion path is determined.

[0068] In the effective human-machine work area, real-time or planned actions are issued through gestures, buttons, voice, etc. Combined with the obtained multi-modal input information, a deep reinforcement learning algorithm is used to generate the running track of the end effector, and the joint space combination sequence required by the physical robot is calculated through the inverse kinematics solver. The process of calculating the joint space combination sequence required by the physical robot through the inverse kinematics solver includes: establishing a robot model, defining a target position and attitude sequence, calculating the joint angle corresponding to each target point using inverse kinematics principles, generating a joint space combination sequence, ensuring that the robot can accurately perform the task path, including but not limited to multi-modal input analysis, deep reinforcement learning algorithm principles and inverse kinematics principle solving. Then map the joint space combination sequence to the joint space of the virtual robot, thereby realizing the path planning of the following operation.

[0069] In a preferred embodiment, in order to solve the problem of robot motion generation in industrial complex tasks, the motion planning method based on deep reinforcement learning includes the following steps: first, according to the operation requirements of the operator, set the target 6D attitude or motion track of the virtual robot to ensure the accuracy and executability of the task target; then, according to the current attitude and target attitude of the virtual robot, determine the running speed and use the neural network controller generated by the deep reinforcement learning combined with the inverse kinematics solver to calculate the joint value of the virtual robot under the target attitude, thereby realizing efficient and accurate attitude control. Then, using the spatial computing power and visualization capability of mixed reality, check whether there is a potential collision possibility in the running process. Then, combined with the attitude conversion matrix and calibration data of the virtual robot, the attitude of the physical robot is calculated to ensure accurate alignment between the virtual and physical robots. In the motion planning based on deep reinforcement learning, the input of the deep reinforcement learning motion planning module is the target attitude of the virtual robot in the mixed reality environment, and the output is the joint state of the robot under the target attitude calculated by the inverse kinematics solver. Through the attitude conversion configuration module, these joint state outputs are output as the joint state of the physical robot, including the base joint, shoulder joint, elbow joint, wrist 1 joint, wrist 2 joint and wrist 3 joint. Finally, these joint states are output to the physical robot controller to ensure that the robot can complete the task according to the predetermined track and attitude, realizing efficient task execution and accurate motion generation. Through the above steps, the present application provides a robot safe motion planning method combined with deep reinforcement learning, which can effectively solve the motion generation problem in industrial complex tasks and improve the flexibility and precision of robot operation.

[0070] Step S400: based on the human-robot collaborative motion path, perform human-robot collaborative safety control action.

[0071] Specifically, if the human-robot collaborative motion path does not meet at least one of the preset motion safety standards or the human-robot collaborative control accuracy, a number of new environmental anchor points are reconstructed, and a path planning process is performed based on the new environmental anchor points until the human-robot collaborative motion path meets the preset motion safety standards and the human-robot collaborative control accuracy, and a human-robot collaborative safety control action is performed.

[0072] According to the virtual robot pose and the specific rotation angle and speed value of each joint calculated by inverse kinematics, combined with the distance space calculation of the mixed reality device, the Cartesian coordinates of the robot can ensure that the trajectory generated by the robot does not exceed the effective working range. If it does not exceed, the physical robot joint space transformation can be triggered. The operations include safety distance calculation, trajectory verification and joint space conversion.

[0073] Determine whether the human-robot collaborative motion path meets the robot motion safety requirements and execution accuracy, the mixed reality device calculates the reconstruction of the environment and checks whether the robot digital twin and the reconstructed environment collide, if not, it meets the non-collision safety specification, supplemented by human subjective inspection. If not, the human-robot collaborative motion path planning is re-performed. If the planned human-robot collaborative motion path meets the robot execution accuracy in the spatial coordinate system within the safety range and there is no potential collision danger, the calculated joint command can drive the physical robot to perform motion in the workshop, complete the human-robot bidirectional safety collaborative feedback control.

[0074] In summary, the beneficial effects of the present application include:

[0075] The present application can capture the multi-modal input of human operators in real time (such as gestures, speech, touch, etc.), human-machine distance and scene information, thereby providing immediate and accurate feedback for human-machine collaboration scenarios. The virtual robot in the mixed reality device provides multi-modal control input for robot controllers, including tactile, visual and audio feedback, which not only improves the intuitiveness and convenience of operation, but also enhances the efficiency and accuracy of complex human-robot collaborative work. The seamless real-time data transmission between the physical robot and the virtual robot is realized through the digital twin communication mechanism, ensuring the real-time nature of the robot operation. At the same time, through the pose registration matrix, the physical robot can accurately reproduce the pre-planned trajectory, ensuring high precision and consistency of the operation.

[0076] The method of the present application has high adaptability and universality, and can adapt to various different manufacturing scenes and tasks. Whether it is different characters, complex scenes or diversified task requirements, the system can provide effective safety protection. Its modular design allows the system to be flexibly configured and expanded in different application scenarios, thereby meeting the human-machine collaboration requirements in different industrial environments. Through intelligent perception and analysis of the task and environment, the system can dynamically adjust its working mode and safety strategy to adapt to the real-time changes of the operating environment and task requirements.

[0077] The present application decouples the safety protection measures into progressive and multi-level strategies. This design not only avoids the problem of insufficient protection and precision decline caused by a single module in the deployment process, but also improves the overall safety and reliability of the system. Reinforcement learning motion planning based on inverse kinematics enables virtual robots to intuitively accept and understand the flexible operating poses of robot controllers. Through inverse calculation, the system can accurately calculate the joint values of the robot at the target pose and pass them to the physical robot, ensuring that it can perform precise and flexible task operations. In addition, the system also integrates multi-sensor data fusion technology to improve the accuracy of environmental perception and task execution, thereby further improving the safety and efficiency of robot operation.

[0078] As shown in FIG. 3, corresponding to the above-mentioned human-machine collaboration safety control method based on mixed reality and digital twin, the embodiment of the present application also provides a human-machine collaboration safety control device based on mixed reality and digital twin, the human-machine collaboration safety control device based on mixed reality and digital twin comprises:

[0079] The information acquisition module 310 is configured to acquire first pose information of a physical robot and second pose information of a virtual robot.

[0080] The registration module 320 is configured to perform pose registration on the physical robot and the virtual robot based on the first pose information and the second pose information, and obtain a registration result matrix.

[0081] The human-machine collaborative path planning module 330 is configured to perform path planning using reinforcement learning based on the registration result matrix and a preset human-machine working area, and determine a human-machine collaborative motion path.

[0082] The control execution module 340 is configured to perform human-machine collaboration safety control actions based on the human-machine collaborative motion path.

[0083] Specifically, in the present embodiment, the specific functions of the human-machine collaboration safety control device based on mixed reality and digital twin can also be referred to the corresponding description in the human-machine collaboration safety control method based on mixed reality and digital twin, which will not be repeated here.

[0084] Based on the above-mentioned embodiments, the application further provides an intelligent terminal, and a principle block diagram of the intelligent terminal can be shown in FIG. 4. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a human-machine cooperation safety control program based on mixed reality and digital twin. The internal memory provides an environment for the operating system and the human-machine cooperation safety control program based on mixed reality and digital twin in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with external terminals through network connection. The human-machine cooperation safety control program based on mixed reality and digital twin realizes the steps of any one of the human-machine cooperation safety control methods based on mixed reality and digital twin when executed by the processor. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0085] Those skilled in the art can understand that the principle block diagram shown in FIG. 4 is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the intelligent terminal to which the application scheme is applied. The specific intelligent terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0086] In one embodiment, an intelligent terminal is provided, and the above-mentioned intelligent terminal includes a memory, a processor and a human-machine cooperation safety control program based on mixed reality and digital twin stored on the above-mentioned memory and executable on the above-mentioned processor. The human-machine cooperation safety control program based on mixed reality and digital twin realizes the steps of any one of the human-machine cooperation safety control methods based on mixed reality and digital twin provided by the application embodiment when executed by the processor.

[0087] The application embodiment further provides a computer readable storage medium, and the computer readable storage medium stores a human-machine cooperation safety control program based on mixed reality and digital twin. The human-machine cooperation safety control program based on mixed reality and digital twin realizes the steps of any one of the human-machine cooperation safety control methods based on mixed reality and digital twin provided by the application embodiment when executed by the processor.

[0088] It should be understood that the sequence numbers of the steps in the above-mentioned embodiments do not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the application embodiment.

[0089] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above-mentioned system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0090] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0091] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different ways to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0092] In the embodiments provided by the present application, it should be understood that the disclosed device / terminal equipment and method can be implemented by other ways. For example, the above-mentioned device / terminal equipment embodiments are only schematic, for example, the division of the above-mentioned modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0093] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand; the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not deviate from the spirit and scope of the corresponding technical solutions, and should be included in the protection scope of the present application.

Claims

1. A human-machine collaboration safety control method based on mixed reality and digital twin, characterized in that, The method comprises the following steps: obtaining first pose information of a physical robot and second pose information of a virtual robot; based on the first pose information and the second pose information, performing pose registration on the physical robot and the virtual robot to obtain a registration result matrix; based on the registration result matrix and a preset human-machine working area, performing path planning by using reinforcement learning to determine a human-robot collaborative motion path; based on the human-robot collaborative motion path, performing human-robot collaborative safety control actions.

2. The human-machine collaboration safety control method based on mixed reality and digital twin according to claim 1, characterized in that, The method comprises the following steps: based on the first pose information and the second pose information, calculating a pose registration matrix; based on the pose registration matrix, performing pose registration and coordinate system registration on the physical robot and the virtual robot to obtain a registration result matrix.

3. The human-machine collaboration safety control method based on mixed reality and digital twin according to claim 1, characterized in that, The method comprises the following steps: based on a preset human-robot collaborative control accuracy, constructing a plurality of environmental anchor points; based on all the environmental anchor points and the human-machine working area, calculating the distance between the physical robot and the virtual robot; based on the distance, determining the linear velocity of the physical robot motion; based on the linear velocity, deep reinforcement learning principles and inverse kinematics principles, performing path planning to determine a human-robot collaborative motion path.

4. The human-machine collaboration safety control method based on mixed reality and digital twin according to claim 3, characterized in that, The method comprises the following steps: based on a preset communication frequency, performing periodic polling on the collaborative working state of the physical robot and the virtual robot to determine the current collaborative working state; obtaining the current environmental anchor point, and based on the current collaborative working state and the current environmental anchor point, determining the current human-robot collaborative control accuracy; if the current human-robot collaborative control accuracy meets the preset human-robot collaborative control accuracy, constructing a plurality of environmental anchor points.

5. The human-machine collaboration safety control method based on mixed reality and digital twin according to claim 3, characterized in that, The method comprises the following steps: based on the linear velocity, deep reinforcement learning principles and inverse kinematics principles, calculating joint angular velocity, and using the joint angular velocity and a preset target pose to generate the motion trajectory of the physical robot; based on the motion trajectory of the physical robot, calculating the joint space combination sequence of the physical robot; mapping the joint space combination sequence to the joint space of the virtual robot to determine the joint space combination sequence of the virtual robot; based on the joint space combination sequence of the virtual robot, determining the motion path of the virtual robot; based on the motion trajectory of the physical robot and the motion path of the virtual robot, determining a human-robot collaborative motion path.

6. The human-machine collaboration safety control method based on mixed reality and digital twin according to claim 5, characterized in that, The method comprises the following steps: based on the joint space combination sequence of the virtual robot, calculating the joint motion value of the virtual robot; Determine a motion path of the virtual robot based on the joint motion value and the human-robot work area.

7. The human-machine collaboration safety control method based on mixed reality and digital twin according to claim 3, characterized in that, Perform a human-robot collaborative safety control action based on the human-robot collaborative motion path. If the human-robot collaborative motion path does not meet at least one of the preset motion safety standard or the human-robot collaborative control accuracy, reconstruct a plurality of new environmental anchor points, and perform a path planning process based on the new environmental anchor points until the human-robot collaborative motion path meets the preset motion safety standard and the human-robot collaborative control accuracy, and perform a human-robot collaborative safety control action.

8. A human-machine collaboration safety control device based on mixed reality and digital twin, characterized in that, The device comprises: An information acquisition module configured to acquire first pose information of a physical robot and second pose information of a virtual robot; A registration module configured to perform pose registration on the physical robot and the virtual robot based on the first pose information and the second pose information to obtain a registration result matrix; A human-robot collaborative path planning module configured to perform path planning using reinforcement learning based on the registration result matrix and a preset human-robot work area to determine a human-robot collaborative motion path; A control execution module configured to perform a human-robot collaborative safety control action based on the human-robot collaborative motion path.

9. An intelligent terminal, characterized by The intelligent terminal comprises a memory, a processor, and a human-robot collaborative safety control program based on mixed reality and digital twinning stored on the memory and executable on the processor, and the human-robot collaborative safety control program based on mixed reality and digital twinning, when executed by the processor, implements the steps of the human-robot collaborative safety control method based on mixed reality and digital twinning of any one of claims 1-7.

10. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a human-robot collaborative safety control program based on mixed reality and digital twinning, and the human-robot collaborative safety control program based on mixed reality and digital twinning, when executed by the processor, implements the steps of the human-robot collaborative safety control method based on mixed reality and digital twinning of any one of claims 1-7.

Images