Virtual group interaction system, method, terminal device and storage medium

Abstract

This application relates to the field of virtual interaction technology and discloses a virtual group interaction system, method, terminal device, and storage medium. The system includes multiple robots and a control platform that runs a virtual environment. The control mode of each human-machine controller is determined according to the mechanical characteristics of the robot. The control mode of the platform controller is determined according to the modeling method of the virtual objects in the virtual environment. Each human-machine controller is used to connect to a robot for user control, so that when a motion signal from the user acting on the robot end is detected, the robot end is driven to react to the user according to the virtual signal sent by the platform controller. The platform controller is used to receive the motion signal from each robot end and, after calculating the motion signal, control the movement of the same virtual object or different virtual objects in the virtual environment to achieve virtual group interaction. This application is based on the tactile mediation of multiple robots and can realistically reproduce group interaction tasks in real life.

Description

Technical Field

[0001] The present application relates to the field of virtual interaction technology, and in particular to a virtual group interaction system, method, terminal device and storage medium. Background Art

[0002] Virtual reality (VR) interaction technology is being used more and more widely in many fields. Among them, the existing virtual interaction technology is often one-way. For example, users can only perform some one-way control operations in the virtual environment through the corresponding virtual devices; or the interaction is limited to the characters in the virtual environment. For each user, it is mainly to simply see visually that the character they control can interact with other characters in the virtual environment, while the user himself cannot feel the real human-computer interaction force from other users. Therefore, the implementation of only one-way control in virtual interaction technology will lead to the inability to effectively perceive the amount of force used by oneself, and the inability to feedback and perceive the amount of force used by multiple people during multi-person interaction, resulting in a poor user experience. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a virtual group interaction system, method, terminal device and storage medium, which can effectively solve the problem of poor user experience during virtual group interaction.

[0004] In a first aspect, embodiments of the present application provide a virtual group interaction system, comprising: a plurality of robots and a control platform running a virtual environment; wherein the control mode of each human-machine controller is determined according to the mechanical characteristics of the robots; and the control mode of the platform controller is determined according to the modeling method of virtual objects in the virtual environment;

[0005] Each of the human-machine controllers is used to connect to a robot for user control, so that when a motion signal exerted by the user on the end of the robot is detected, the end of the robot is driven to react to the user according to a virtual signal sent by the platform controller;

[0006] The platform controller is used to receive motion signals from each of the robot terminals, and to control the movement of the same virtual object or different virtual objects in the virtual environment after calculating the motion signals, so as to realize virtual group interaction.

[0007] In some embodiments, the types of control modes include admittance mode and impedance mode; when the control modes of the human-machine controller and the platform controller are the same, the virtual group interaction system also includes: an intermediate controller; the intermediate controller is used for signal conversion processing between the human-machine controller and the platform controller.

[0008] In some embodiments, when the human-machine controller and the platform controller are both configured in admittance mode, the intermediate controller is configured in impedance mode, and is configured to convert the end displacement signal in the motion signal output by the human-machine controller and output it to the platform controller;

[0009] Alternatively, when the human-machine controller and the platform controller are both configured in impedance mode, the intermediate controller is configured in admittance mode, and is used to convert the human-machine interaction force in the motion signal output by the human-machine controller and output it to the platform controller.

[0010] In some embodiments, when the human-machine controller is configured in admittance mode and the platform controller is configured in impedance mode, the end displacement signal in the motion signal output by the human-machine controller is directly sent to the platform controller;

[0011] Alternatively, when the human-machine controller is configured in impedance mode and the platform controller is configured in admittance mode, the human-machine interaction force in the motion signal output by the human-machine controller is directly sent to the platform controller.

[0012] In some embodiments, the platform controller sends the virtual signal to each of the human-machine controllers via a distributed network;

[0013] Each human-machine controller is provided with a local virtual object. After receiving the virtual signal, the virtual object is compliantly controlled so as to follow the virtual object in the virtual environment.

[0014] In a second aspect, an embodiment of the present application provides a virtual group interaction method applicable to a human-machine controller in the above-mentioned virtual group interaction system, the virtual group interaction method comprising: acquiring a motion signal of a robot end and sending the motion signal to the platform controller;

[0015] receiving a virtual signal output by the platform controller, and processing the virtual signal and the motion signal to obtain a motor control signal of the robot;

[0016] The robot is driven to move according to the motor control signal, so as to generate an interaction signal at the end of the robot that reacts to the user.

[0017] In a second aspect, an embodiment of the present application provides a virtual group interaction method, applicable to the platform controller in the above-mentioned virtual group interaction system, the virtual group interaction method comprising:

[0018] Receive the motion signals of the robot end sent by each human-machine controller;

[0019] Detect whether the virtual objects acted upon by each user through the robot are the same, calculate the virtual signal corresponding to the virtual object based on the motion signal, and control the movement of the same virtual object or different virtual objects in the virtual environment based on the virtual signal to achieve virtual group interaction.

[0020] In some embodiments, when the motion signal is the end displacement information of the robot, the virtual signal calculated based on the end displacement information is a virtual force, or a virtual force and a virtual displacement signal;

[0021] Alternatively, when the motion signal is a human-computer interaction force, the virtual signal obtained according to the human-computer interaction force is a virtual displacement signal, or a virtual displacement signal and a virtual force.

[0022] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the above-mentioned virtual group interaction method.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the above-mentioned virtual group interaction method when executed on a processor.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] This application determines the control mode of the corresponding human-machine controller based on the mechanical characteristics of each robot, so that different robots are configured with appropriate control modes, thereby achieving independent and precise control of each robot. Furthermore, when multiple users use robots with compatible human-machine controllers as tactile mediators to interact with the platform controller of the virtual environment in a group, each user can feel the force exerted by other users, greatly increasing the sense of reality during multi-user operation. At the same time, the platform controller also determines its compatible platform controller based on the modeling method of virtual objects in the virtual environment. Through information exchange between all controllers, all independent terminals in the virtual group interaction system can interact accurately. That is, based on the robot's tactile mediation, it can truly restore the cooperative (or competitive) group interaction tasks in real life. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A schematic diagram of the structure of a virtual group interaction system according to an embodiment of the present application is shown;

[0028] Figure 2 A first model diagram of the virtual group interaction system according to an embodiment of the present application is shown;

[0029] Figure 3 A schematic diagram showing a first principle structure of a virtual group interaction system according to an embodiment of the present application is shown;

[0030] Figure 4 A second model diagram of the virtual group interaction system according to an embodiment of the present application is shown;

[0031] Figure 5 A second schematic diagram of the principle structure of the virtual group interaction system according to an embodiment of the present application is shown;

[0032] Figure 6 A third model diagram of the virtual group interaction system according to an embodiment of the present application is shown;

[0033] Figure 7 A third schematic diagram of the structure of the virtual group interaction system according to an embodiment of the present application is shown;

[0034] Figure 8 A fourth model diagram of the virtual group interaction system according to an embodiment of the present application is shown;

[0035] Figure 9 A fourth principle structural diagram of the virtual group interaction system according to an embodiment of the present application is shown;

[0036] Figure 10 A schematic diagram of a distributed network architecture of a virtual group interaction system according to an embodiment of the present application is shown;

[0037] Figure 11 A flow chart of a virtual group interaction method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0039] The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0040] Hereinafter, the terms "including", "having" and their cognates used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the aforementioned items, and should not be understood as excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the aforementioned items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the aforementioned items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and should not be understood as indicating or implying relative importance.

[0041] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0042] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0043] The embodiment of the present application proposes a virtual group interaction system, such as Figure 1 As shown, the virtual group interaction system includes a control platform capable of operating a virtual environment and multiple robots connected to the virtual group interaction system. Assuming that the user's contact with the robot does not cause relative displacement, each robot acts as a tactile intermediary between the corresponding user and the virtual environment. Users can manipulate the robot to perform various motion interactions on virtual objects in the virtual environment. Each user can also feel the virtual force acting on them, thus achieving tactile feedback. It is also understandable that when multiple users simultaneously act on virtual objects in the virtual environment, each user can feel the force of the other users, thus achieving tactile group interconnection. This can greatly enhance the realism of multi-user operations and make them more interesting.

[0044] The virtual group interaction method is described below with reference to some specific embodiments.

[0045] Figure 1A schematic diagram of the structure of a virtual group interaction system according to an embodiment of the present application is shown. Exemplarily, the virtual group interaction system includes: multiple robots and a control platform running a virtual environment; the control mode of each human-machine controller is determined based on the mechanical characteristics of each robot; each human-machine controller is connected to a user-controlled robot, and upon detecting a user's action on the robot's end, drives the robot's end and reacts to the user controlling the robot based on a virtual signal sent by the platform controller; the control mode of the platform controller in the control platform is determined based on the modeling method of virtual objects in the virtual environment; the platform controller is configured to receive motion signals from the robot's end sent by each human-machine controller, calculate the motion signals, and then control the movement of the same virtual object or different virtual objects in the virtual environment to achieve virtual group interaction.

[0046] This application determines the control mode of the human-machine controller based on the mechanical characteristics of each robot, so that the human-machine controllers corresponding to different robots are configured with appropriate control modes, thereby improving the accuracy of robot control; at the same time, the control mode of the platform controller in the virtual platform is also determined according to the modeling method of the virtual object; all human-machine controllers and platform controllers in the virtual group interaction system independently and accurately determine their respective adaptive control modes. Therefore, the entire virtual group interaction system has better flexibility and control accuracy, and can truly restore the cooperative (or competitive) group interaction tasks in real life, greatly increasing the sense of reality during multi-user operation.

[0047] like Figure 1 As shown, multiple users (User 1, User 2, ..., User n) interact with the platform controller of the control platform by manipulating the corresponding robot in the virtual group interaction system, enabling group interaction between the users and the same virtual object or different virtual objects in the virtual environment. Therefore, User 1 not only experiences tactile feedback from the same virtual object or different virtual objects in the virtual environment, but also experiences direct or indirect effects from User 2, ..., and User n. For example, in a multi-player tug-of-war game, each user can feel the combined force exerted on the rope by all other users. Because multiple users act on the rope simultaneously, each user can directly feel the direct effects of the other users. For example, in a multi-player table tennis game, each user experiences tactile feedback related to the ball's motion state when in contact with the ball. The ball exhibits a unique motion state as a result of the effects of different users acting on it. Therefore, the subsequent user indirectly feels the indirect effects of the previous user.

[0048] The motion characteristics of natural objects are often the conversion of force into displacement, such as in a spring: F = kx and Newton's second law: F = ma. In some control systems, the control system types can be categorized into admittance mode and impedance mode based on the principle of the control method. The admittance mode has a force input and a displacement output; the impedance mode has a displacement input and a force output. In virtual group interaction, properly configuring the control system can improve the accuracy of independent control by each terminal during group interaction control.

[0049] In the specific implementation of this application, the control mode of each human-machine controller is determined according to the mechanical characteristics of the robot. One specific implementation method is: the control mode of the human-machine controller for controlling the robot is determined according to the size of the robot's inertia; when the robot's inertia is large, it is necessary to control the robot's displacement signal, then the type of control mode of the human-machine controller is the impedance mode; when the robot's inertia is small, it is necessary to control the robot's force signal, then the type of control mode of the human-machine controller is the admittance mode. As another implementation method, the control mode of the human-machine controller can also be determined based on the reduction ratio; this application does not limit the basis for determining the type of control mode of the human-machine controller.

[0050] In the specific implementation of this application, the control mode of the platform controller in the control platform running the virtual environment is determined based on the modeling method of the virtual objects in the virtual environment. One specific implementation method is: the control mode of the corresponding platform controller is determined based on the hardness characteristics of the virtual objects in the virtual environment; when the virtual object is a rigid object, it is necessary to control the virtual object with a force signal, and the platform controller type of the virtual environment is an admittance mode; when the virtual object is an elastic object, it can be controlled with either a displacement signal or a force signal, and the platform controller type of the virtual environment is an impedance mode or an admittance mode.

[0051] In a virtual group interaction system, it's assumed that a user's hand controls the movement of a robot's handle. The user then uses the robot as a tactile medium to interact with virtual objects in the virtual environment. During this virtual group interaction, it's assumed that there's no relative displacement between the user's hand and the robot's handle. The following describes the group interactions generated by this virtual group interaction system using different system models and their corresponding structures.

[0052] When the control mode of the human-machine controller and the control mode of the platform controller are both admittance modes, the first model diagram of the virtual group interaction system is as follows: Figure 2As shown; the virtual group interaction system also includes an intermediate controller configured in impedance mode; the intermediate controller in impedance mode is connected between the human-machine side (user and robot) and the virtual environment, and the intermediate controller is used to realize the bridge conversion processing between the displacement signal output by the human-machine controller and the force signal input by the platform controller; this is because after the platform controller is configured in admittance mode, it needs to input force signals and output displacement signals; and the human-machine controller on the human-machine side is also configured in admittance mode, and at this time it also needs to input force signals and output displacement signals; in order to make the platform controller and the human-machine controller on the human-machine side perform friendly information interaction, it is necessary to use the intermediate controller in impedance mode as an auxiliary bridge to perform signal conversion processing between the human-machine controller and the platform controller, that is, to match the information interaction between the platform controller and the human-machine controller through the intermediate controller in impedance mode.

[0053] The first model of the virtual group interaction system is introduced as an example. Assuming that the control mode of the human-machine controller and the control mode of the platform controller are both configured as the admittance mode, the first principle structure diagram of the corresponding virtual group interaction system is as follows: Figure 3 As shown, there are three users on the human-machine side. The control mode of the human-machine controller corresponding to the robot touched by each user is configured in the admittance mode. Each human-machine side will detect the end displacement signal x of the robot through sensors. h1 、x h2 、x h3 They are sent to the intermediate controller corresponding to the impedance mode, and the intermediate controller converts the robot's end displacement signal x h1 、x h2 、x h3 and the virtual displacement signal x of the virtual object v1 、x v2 、x v3 Perform the transformation to obtain the corresponding virtual force F v1 、F v2 、F v3 In this example, the impedance model configured by the impedance mode of the intermediate controller is:

[0054] x ei =x hi -x vi

[0055]

[0056] Where x hi is the end displacement signal of the robot under the action of the i-th user; x vi is the virtual displacement signal of the virtual object under the action of the i-th user, and its initial displacement is 0; K and B are the inertia and damping parameters of the impedance model.

[0057] This application does not limit the specific implementation method of the impedance model. In this example, the impedance model is related to the end displacement signal of the robot, the virtual displacement signal of the virtual object, and the first-order derivative (velocity) of the relative displacement difference; as other implementation methods, it can also be related to the second-order derivative (acceleration) of the relative displacement difference.

[0058] In one embodiment, the platform controller controls each virtual force F according to the configuration of the virtual environment. v1 、F v2 、F v3 Calculate and process the virtual displacement signal x of the virtual object vi At the same time, each virtual force F v1 、F v2 、F v3 Send it to the corresponding human-machine controller, so that the human-machine controller can respond to the virtual force F according to the feedback. v1 、F v2 、F v3 The corresponding motor control signal τ is obtained by processing the human-machine interaction forces F1, F2, and F3 in the motion signal of the robot applied by the user c1 , τ c2 , τ c3 ; Among them, each motor control signal is a vector signal, and the dimension of the vector signal is determined by the number of robot joints; the human-machine controller controls the motor torque of each joint, so that the reaction force at the end of the robot is the virtual force F v1 、F v2 、F v3 , allowing each user to experience the virtual force of the virtual object's reaction. During this process, the platform controller sends the virtual object's virtual displacement signal to the intermediate controller to update the virtual object's virtual displacement signal in real time, allowing the intermediate controller to calculate and update the virtual force in real time, thereby allowing each user to feel the latest virtual force feedback from the virtual object more realistically and promptly.

[0059] When the platform controller calculates the virtual displacement signal of the virtual object, an exemplary calculation method is to first build a physical model for simulation, wherein the model of the mass block with the damping term Through this exemplary model, a virtual displacement signal of a virtual object can be calculated. It should be noted that any model that can calculate a virtual displacement signal of a virtual object can be used, and this application does not limit the specific structure of the model.

[0060] Specifically, when user 1 interacts with the virtual objects in the virtual environment through robot 1, the human-machine interaction force exerted by user 1 on robot 1 is F1, and the end displacement signal of robot 1 is x h1 , the end displacement signal x of robot 1 h1Send to the intermediate controller, the intermediate controller will calculate the end displacement information x of robot 1 h1 and the virtual displacement signal x of the virtual object vi Perform the conversion process to obtain the virtual force F corresponding to user 1 v1 Similarly, the virtual force F corresponding to user 2 can be obtained v2 The virtual force F corresponding to user 3 v3 ; The virtual environment applies three virtual forces F according to the configuration of the virtual environment v1 、F v2 、F v3 Processing is performed to obtain the virtual displacement signal x of the virtual object v1 , and the virtual displacement signal x of the virtual object v1 is sent to the intermediate controller, thereby adjusting the virtual displacement signal x of the virtual object in the intermediate controller v1 At the same time, the intermediate controller updates the virtual force F corresponding to user 1 v1 Feedback is sent to the human-machine end, and the human-machine controller at the human-machine end responds to the input desired instruction, that is, the feedback virtual force F v1 The human-machine interaction force F1 exerted by user 1 on the end of robot 1 is processed to obtain the corresponding motor control signal τ c1 , the robot 1 controls the motor torque of each joint through its human-machine controller 1, so that the feedback force at the end of the robot 1 is the virtual force F v1 , that is, the virtual force F that user 1 feels during the interaction with the virtual environment v1 , and so on, user 2 feels the virtual force F during the interaction with the virtual environment v2 , user 3 feels the virtual force F during the interaction with the virtual environment v3 Furthermore, during the process of continuous interaction of each user in the virtual environment, the corresponding virtual force can be updated and fed back to each user in real time based on the updated virtual displacement signal of the virtual object.

[0061] When the control mode of the human-machine controller is configured as the admittance mode and the control mode of the platform controller is configured as the impedance mode, the corresponding second model diagram of the virtual group interaction system is as follows: Figure 4 As shown in the figure; when the control mode of the platform controller is configured in impedance mode, it is necessary to input a displacement signal and output a force signal; and when the control mode of the human-machine controller on the human-machine side is configured in admittance mode, it is necessary to input a force signal and output a displacement signal; the force signal output by the platform controller and the force signal required to be input by the human-machine controller, and the displacement signal output by the human-machine controller and the displacement signal required to be input by the platform controller just match each other. Therefore, the platform controller and the human-machine controller in the virtual group interaction system can directly receive and process signals.

[0062] The second model of the virtual group interaction system is introduced as an example. Assuming that the control modes of the human-machine controllers of multiple human-machine terminals are configured in the admittance mode and the control mode of the platform controller is configured in the impedance mode, the corresponding second principle structure diagram of the virtual group interaction system is as follows: Figure 5 As shown in the figure, the user, robot and controller are the human-machine side, and the three users on the human-machine side respectively convert the robot's end displacement signal x h1 、x h2 、x h3 Sent to the control platform of the virtual environment, the platform controller calculates the virtual forces F according to the configuration of the virtual environment v1 、F v2 、F v3 ; Then the virtual force F v1 、F v2 、F v3 The human-machine controller at the corresponding human-machine end sends the virtual force F to the human-machine controller at the corresponding human-machine end. v1 、F v2 、F v3 The corresponding motor control signal τ is obtained by processing the human-machine interaction force F1, F2, and F3 applied by each user to the end of the robot c1 , τ c2 , τ c3 , where each motor control signal is a vector signal, and the dimension of the vector signal is determined by the number of robot joints; each human-machine controller is based on the motor control signal τ c1 , τ c2 , τ c3 Control the motor torque of each joint of each robot so that the reaction force at the end of the robot is the virtual force F v1 、F v2 、F v3 And react to each user through the robot, so that the three users can feel the virtual force F during the interaction with the virtual object. v1 、F v2 、F v3 .

[0063] When the control mode of the human-machine controller is configured as impedance mode and the control mode of the platform controller is configured as admittance mode, the corresponding third model diagram of the virtual group interaction system is as follows: Figure 6As shown in the figure; when the control mode of the platform controller is configured in the admittance mode, it is necessary to input a force signal and output a displacement signal; when the control mode of the human-machine controller is configured in the impedance mode, it is necessary to input a displacement signal and output a force signal; at this time, the displacement signal output by the platform controller and the displacement signal required to be input by the human-machine controller, and the force signal output by the human-machine controller and the force signal required to be input by the platform controller just match each other. Therefore, the platform controller and the human-machine controller in the virtual group interaction system can directly receive and process signals.

[0064] The third model of the virtual group interaction system is introduced as an example. Assuming that the control modes of the human-machine controllers of multiple human-machine terminals are configured in the admittance mode and the control mode of the platform controller is configured in the impedance mode, the corresponding third principle structure diagram of the virtual group interaction system is as follows: Figure 7 As shown in the figure, the user, robot and controller are the human-machine side. The three users on the human-machine side will act on the robot's human-machine interaction force F h1 、F h2 、F h3 The platform controller sends the data to the control platform of the virtual environment, and the platform controller controls the human-computer interaction force F according to the configuration of the virtual environment. h1 、F h2 、F h3 Processing is performed to obtain the virtual displacement signal x of the virtual object v1 、x v2 、x v3 ; and the virtual displacement signal x of the virtual object v1 、x v2 、x v3 The human-machine controller at the human-machine end generates the virtual displacement signal x of the virtual object. v1 、x v2 、x v3 The corresponding motor control signal τ is obtained by processing the end displacement signals x1, x2, and x3 applied by the user to the end of the robot. c1 , τ c2 , τ c3 , where each motor control signal is a vector signal, and the dimension of the vector signal is determined by the number of robot joints; each human-machine controller is based on the motor control signals τc1, τ c2 , τ c3 Control the movement of the robot end, the displacement signal of each robot end can be the displacement information x v1 、x v2 、x v3 As another implementation method, the displacement signal at the end of the robot can also be used as speed information

[0065] When the control mode of the human-machine controller and the control mode of the platform controller are configured as impedance mode, the fourth model of the virtual group interaction system is as follows: Figure 8 As shown; the virtual group interaction system also includes an intermediate controller configured with an admittance mode; the intermediate controller in the admittance mode is connected between the human-machine end (user and robot) and the virtual environment, and the intermediate controller is used to realize the bridge conversion processing between the force signal output by the human-machine controller and the displacement signal input by the platform controller; this is because when the platform controller is configured with an impedance mode, it needs to input a displacement signal and output a force signal; and the human-machine controller is also configured with an impedance mode, and at this time it also needs to input a displacement signal and output a force signal; in order to enable the platform controller and the human-machine controller to perform friendly information interaction, it is necessary to use the intermediate controller in the admittance mode as an auxiliary bridge to perform signal conversion processing between the human-machine controller and the platform controller, that is, to match the information interaction between the platform controller and the human-machine controller through the intermediate controller in the admittance mode.

[0066] The fourth model of the virtual group interaction system is introduced as an example. Assuming that the control mode of the human-machine controllers and the control mode of the platform controller of multiple human-machine terminals are both configured in impedance mode, the fourth principle structure diagram of the corresponding virtual group interaction system is as follows: Figure 9 As shown in the figure, the user, robot and controller are the human-machine side. The three users on the human-machine side will act on the robot's human-machine interaction force F h1 、F h2 、F h3 They are sent to the corresponding intermediate controllers, and each intermediate controller sends the human-computer interaction force F h1 、F h2 、F h3 and virtual force F v1 、F v2 、F v3 Perform transformation processing to obtain the virtual displacement signal x of the virtual object v1 、x v2 、x v3 In this example, the admittance mode adopts the first-order admittance model considering the mass-damping link. The first-order admittance mode is:

[0067]

[0068] Where, is the second-order derivative of the virtual displacement information of the virtual object (the acceleration of the virtual object) under the interaction of the i-th user; M e and B e are the inertia and damping parameters of the admittance model respectively; F hi is the human-machine interaction force between the i-th user and the robot; F viis the virtual force felt by the i-th user when touching the virtual object, that is, the virtual force on the virtual object during the interaction process; is the first-order derivative of the virtual displacement of the virtual object (the velocity of the virtual object) under the interaction of the i-th user.

[0069] This application does not limit the specific implementation method of the admittance model. In this example, the admittance model is the first-order model considering the mass-damping link mentioned above. As another implementation method, a second-order model can also be used.

[0070] In a specific embodiment, the platform controller processes the virtual displacement signal output by the intermediate controller according to the configuration of the virtual environment to obtain the virtual forces F v1 、F v2 、F v3 At the same time, the intermediate controller converts the virtual displacement signal x of each virtual object into v1 、x v2 、x v3 Send it to the corresponding human-machine controller, so that the human-machine controller can respond to the feedback virtual displacement signal x v1 、x v2 、x v3 The corresponding motor control signal τ is obtained by processing the end displacement signals x1, x2, and x3 applied by the user to the robot. c1 , τ c2 , τ c3 ; Each motor control signal is a vector signal, and the dimension of the vector signal is determined by the number of robot joints; the human-machine controller responds to the motor control signal τ c1 , τ c2 , τ c3 Feedback control corresponds to the robot motion, so that the displacement signal of each robot end can be the displacement information x v1 、x v2 、x v3 As another implementation method, the displacement signal at the end of the robot can also be the speed information During this process, the platform controller will send the virtual force of the virtual object to the intermediate controller to update the virtual displacement signal of the virtual object in real time, so that each user can feel the virtual displacement signal of the robot end fed back by the virtual object, thereby realizing virtual group interaction.

[0071] Considering that in a real environment, when a user interacts with any object, the object will not only be affected by the corresponding force, but also by environmental forces such as gravity, friction, wind resistance, collision force, elastic force, etc. Therefore, virtual objects in a virtual environment will also have basic physical properties, such as mass, friction coefficient, etc.; in addition, the interaction between the virtual object and the virtual environment also follows basic physical laws, such as Newton's law, momentum theorem, etc. By considering these forces, a more accurate virtual environment force can be calculated. Then, a more accurate virtual force of the virtual object on the user in the virtual environment is calculated, so that the user can have a more realistic tactile feedback. Therefore, in another embodiment, the platform controller adjusts the virtual environment force according to the configuration of the virtual environment. And each virtual force F vi Calculate the virtual displacement information x of the virtual object vi ; and according to each virtual force F vi and the human-machine interaction force F exerted by each user on the end of the robot i Processing is performed to obtain the corresponding motor control signal τ ci , the motor control signal τ ci Under the action of the robot, the motor torque of each joint is controlled, so that each user can feel a more realistic virtual force during the interaction with the virtual environment through the robot.

[0072] Compared with the virtual interaction of a single user, the biggest problem of virtual group interaction is data synchronization and the impact of time delay between different user terminals. Therefore, the virtual group interaction system of this application adopts a distributed network architecture to ensure the security of tactile rendering of each user terminal and timely receive user information from other user terminals to achieve the interconnection of each user's tactile. For example, Figure 10 As shown, for example, the control frequency of the robot of 1000 Hz and the computing rate of the virtual environment simulation are the basic guarantees for ensuring virtual group interaction. Therefore, a distributed network is constructed between the terminal computer device of the running virtual environment, namely the virtual controller, and the terminal computer device of each human-machine end, namely the human-machine controller. Each computer runs independently and communicates directly with other computers to transmit the motion signals of each robot (displacement information and human-machine interaction force); based on the above four models, tactile feedback is provided to the local robot, and the operating state of the virtual object is changed, so as to realize rapid virtual group interaction under the distributed network architecture, ensuring a control frequency of 1000 Hz and low latency.

[0073] To further enhance the synchronization between the tactile experience of the human-machine end and the virtual environment on each computer, a synchronization processing method is adopted: the motion signal of the virtual object is sent to other computers (all human-machine ends) in the distributed network architecture. Each human-machine end uses the received motion signal as a reference motion signal for the virtual object in the virtual environment. At the same time, the human-machine end also has a local virtual object. After receiving the virtual signal (virtual position signal or virtual force) from the platform controller, the human-machine controller performs compliant control on the local virtual object, that is, establishes compliant control between the virtual object and the virtual object, so that the virtual object follows the virtual object. This ensures stable interaction under large delays and also guarantees the maximum fidelity of information during virtual group interaction. In other words, the motion signal of the virtual object in the virtual environment is first sent to the human-machine end. Then, the virtual object set on each human-machine end follows the motion signal of the virtual object, so that the virtual object on each human-machine end directly interacts with the robot instead of the virtual object, thus achieving a synchronized and stable interaction effect. This effectively avoids the calculation asynchrony between the computers in the virtual group interaction caused by communication delay, which leads to significant differences in the virtual objects perceived by different users.

[0074] Applying the above virtual group interaction system, this application also proposes a corresponding virtual group interaction method, such as Figure 11 A flow chart of a virtual group interaction method according to an embodiment of the present application is shown. Exemplarily, the virtual group interaction method includes:

[0075] S100, when each user in the multi-person terminal interacts with a virtual object in a virtual environment by controlling a robot, a motion signal of each robot terminal is obtained.

[0076] Due to the different control modes configured on the HMI controller, the corresponding motion signals obtained from the robot end are different. When the robot's mechanical characteristics on the HMI are high inertia, the HMI controller is configured in impedance mode. In this case, the motion signal obtained from the HMI is the human-machine interaction force applied by the user to the robot. When the robot's mechanical characteristics on the HMI are low inertia, the HMI controller is configured in admittance mode. In this case, the motion signal obtained from the HMI is the displacement information of the robot end. Therefore, the motion signal on the HMI includes both the human-machine interaction force and the displacement information of the robot end.

[0077] S200: The platform controller processes the received motion signal to obtain a virtual signal; and controls the movement of the same virtual object or different virtual objects in the virtual environment according to the virtual signal to achieve virtual group interaction.

[0078] The platform controller of the present application receives motion signals, detects whether the virtual objects acted upon by each user through the robot are the same, calculates the virtual signals corresponding to the virtual objects based on the motion signals, and controls the movement of the same virtual object or different virtual objects in the virtual environment according to the virtual signals to realize virtual group interaction.

[0079] Since the control mode of the human-machine controller is configured as impedance mode or admittance mode, and the control mode of the platform controller is also configured as impedance mode or admittance mode, the virtual group interaction system has different combination modes; specifically, they are as follows:

[0080] When the control mode of the platform controller and the control mode of the human-machine controller are both configured in impedance mode, the intermediate controller in admittance mode is required to convert and process the signal output by the human-machine controller to match the input signal of the platform controller. At this time, the initial signal in the platform controller is the displacement signal of the virtual object; the intermediate controller processes the end displacement signal of the robot according to the initial signal to obtain a virtual signal as the virtual force; the intermediate controller sends the virtual force to the platform controller and the human-machine controller; the platform controller then processes the virtual force to obtain a virtual position signal as the initial signal.

[0081] When the control mode of the platform controller is configured as the admittance mode and the control mode of the human-machine controller is configured as the impedance mode, the platform controller processes the end displacement information of the robot end to obtain a virtual signal as a virtual force.

[0082] When the control mode of the platform controller is configured as the impedance mode and the control mode of the human-machine controller is configured as the admittance mode, the platform controller processes the human-machine interaction force to obtain a virtual displacement signal of the virtual object.

[0083] When the control modes of the platform controller and the human-machine controller are both configured in admittance mode, the intermediate controller in impedance mode is required to convert and process the signal output by the human-machine controller to match the input signal of the platform controller. At this time, the initial signal in the platform controller is the virtual force fed back by the virtual object; the intermediate controller processes the human-machine interaction force according to the initial signal to obtain a virtual signal as a virtual displacement signal, and the intermediate controller sends the virtual displacement signal to the platform controller and the human-machine controller; the platform controller then processes the virtual displacement signal to obtain a virtual force as the initial signal.

[0084] The virtual signal processed by the intermediate controller or the platform controller includes a virtual displacement signal and a virtual force.

[0085] The process of the platform controller processing the virtual signal of the motion information output by the human-machine controller in different combinations in the virtual group interaction system is recorded in detail in the above system embodiment. To avoid repetition, this embodiment only introduces the main technical concepts and will not be described in detail.

[0086] S300, the human-machine controller processes the virtual signal and the motion signal to obtain a motor control signal, and controls the robot end to generate an interactive signal that reacts to the user according to the motor control signal.

[0087] Similarly, since the control mode of the human-machine controller is configured as impedance mode or admittance mode, and the control mode of the platform controller is also configured as impedance mode or admittance mode, the virtual group interaction system has different combination modes; specifically, they are as follows:

[0088] When the control mode of the platform controller and the control mode of the human-machine controller are both configured as impedance mode, the human-machine controller processes the virtual force and the human-machine interaction force to obtain a motor control signal.

[0089] When the control mode of the platform controller is configured as the admittance mode and the control mode of the human-machine controller is configured as the impedance mode, the human-machine controller directly processes the virtual force and the human-machine interaction force to obtain the motor control signal.

[0090] When the control mode of the platform controller is configured as the impedance mode and the control mode of the human-machine controller is configured as the admittance mode, the human-machine controller directly processes the virtual displacement signal and the end displacement information to obtain the motor control signal.

[0091] When the control mode of the platform controller and the control mode of the human-machine controller are both configured as the admittance mode, the human-machine controller processes the virtual displacement signal and the end displacement signal to obtain the motor control signal.

[0092] The process of the human-machine controller processing the virtual signal output by the platform controller or the intermediate controller in different combinations in the virtual group interaction system is recorded in detail in the above system embodiment. To avoid repetition, this embodiment only introduces the main technical concepts and will not be described in detail.

[0093] The implementation method of controlling the robot end to generate an interactive signal that reacts to the user based on the motor control signal is as follows:

[0094] In one embodiment, the robot controls the torque of each joint motor according to the control signal, so that the force at the end of the robot is a virtual force, so that each user can feel the virtual force; that is, the virtual force F that user 1 feels during the interaction with the virtual object is v1 , user 2 feels the virtual force F during the interaction with the virtual objectv2 , user 3 feels the virtual force F during the interaction with the virtual object v3 .

[0095] In another embodiment, the control mode of the human-machine controller controls the movement of the robot according to the control signal feedback, so that the change of the end displacement information of each robot is delayed. The displacement information can be displacement x v1 、x v2 、x v3 As another implementation method, the displacement information of the robot end can also be the speed

[0096] It can be understood that the method of this embodiment corresponds to the group interaction method in the virtual interaction system in the above embodiment. The optional items in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0097] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program, thereby enabling the terminal device to execute the above-mentioned virtual group interaction method.

[0098] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU) and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0099] The memory may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0100] The present application also provides a computer-readable storage medium storing a computer program that, when executed on a processor, implements the aforementioned virtual group interaction method. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0102] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0103] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0104] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A virtual group interaction system, characterized in that: include: A plurality of robots and a control platform operating a virtual environment are constructed as a distributed network; wherein the control mode of each human-machine controller is determined according to the mechanical characteristics of the robot; the control mode of the platform controller is determined according to the modeling method of virtual objects in the virtual environment; the types of control modes include admittance mode and impedance mode; when the control modes of the human-machine controller and the platform controller are the same, the virtual group interaction system further includes an intermediate controller; the intermediate controller is used for signal conversion processing between the human-machine controller and the platform controller; Each of the human-machine controllers is used to connect to a robot for user control, so that when a motion signal exerted by the user on the robot end is detected, the robot end is driven to react to the user according to a virtual signal sent by the platform controller; wherein the motion signal includes human-machine interaction force and displacement information of the robot end, and the displacement information includes displacement or velocity; The platform controller is configured to receive motion signals from each of the robot terminals, detect whether the virtual objects acted upon by the robots by the respective users are the same, calculate virtual signals corresponding to the virtual objects based on the motion signals, and control the movement of the same virtual object or different virtual objects in the virtual environment based on the virtual signals to achieve virtual group interaction; and transmit the virtual signals to each of the human-machine controllers via the distributed network; Each human-machine controller is provided with a local virtual object. After receiving the virtual signal, the virtual object is compliantly controlled so as to follow the virtual object in the virtual environment.

2. The virtual group interaction system according to claim 1, characterized in that: When the human-machine controller and the platform controller are both configured in admittance mode, the intermediate controller is configured in impedance mode, and is used to convert the end displacement signal in the motion signal output by the human-machine controller and output it to the platform controller; Alternatively, when the human-machine controller and the platform controller are both configured in impedance mode, the intermediate controller is configured in admittance mode, and is used to convert the human-machine interaction force in the motion signal output by the human-machine controller and output it to the platform controller.

3. The virtual group interaction system according to claim 1, characterized in that: When the human-machine controller is configured in the admittance mode and the platform controller is configured in the impedance mode, the end displacement signal in the motion signal output by the human-machine controller is directly sent to the platform controller; Alternatively, when the human-machine controller is configured in impedance mode and the platform controller is configured in admittance mode, the human-machine interaction force in the motion signal output by the human-machine controller is directly sent to the platform controller.

4. A virtual group interaction method, characterized in that: The human-machine controller in the virtual group interaction system according to any one of claims 1 to 3, wherein the virtual group interaction method comprises: Acquire the motion signal of the robot end and send it to the platform controller; receiving a virtual signal output by the platform controller, and processing the virtual signal and the motion signal to obtain a motor control signal of the robot; The robot is driven to move according to the motor control signal, so as to generate an interaction signal at the end of the robot that reacts to the user.

5. A virtual group interaction method, characterized in that: A platform controller applicable to the virtual group interaction system according to any one of claims 1 to 3, wherein the virtual group interaction method comprises: Receive the motion signals of the robot end sent by each human-machine controller; Detect whether the virtual objects acted upon by each user through the robot are the same, calculate the virtual signal corresponding to the virtual object based on the motion signal, and control the movement of the same virtual object or different virtual objects in the virtual environment based on the virtual signal to achieve virtual group interaction.

6. The virtual group interaction method according to claim 5, characterized in that: When the motion signal is the end displacement information of the robot, the virtual signal calculated based on the end displacement information is a virtual force or a virtual force and a virtual displacement signal; Alternatively, when the motion signal is a human-computer interaction force, the virtual signal obtained according to the human-computer interaction force is a virtual displacement signal or a virtual displacement signal and a virtual force.

7. A terminal device, characterized in that: The terminal device includes a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the virtual group interaction method according to any one of claims 4 to 6.

8. A computer-readable storage medium, characterized in that The device stores a computer program, which, when executed on a processor, implements the virtual group interaction method according to any one of claims 4 to 6.

Images