A digital twin interaction method, device, equipment and medium

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a digital twin interaction method, device, equipment and medium, which are applied to a digital twin interaction system, wherein the system includes a tactile device, a virtual object, a dynamic solution servo and a force feedback output servo; including: obtaining the position change of the tactile device in the sampling period when interacting with the virtual object; generating an estimated position of the tactile device based on the position change, the lag time of the dynamic solution servo and the smoothing time of the force feedback output servo; determining the estimated feedback force corresponding to the estimated position; making an error judgment on the estimated feedback force, and determining the transition feedback force according to the judgment result; converting the transition feedback force according to a preset mapping coefficient to obtain the target feedback force of the virtual object on the tactile device. Its beneficial effect is to achieve coupling and synchronization between servos and improve the coherence of the interaction effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a digital twin interaction method, device, equipment and medium. Background Art

[0002] The application of digital twin technology in the health management of marine electromechanical equipment has become a key means to improve the efficiency and safety of equipment maintenance. By building a virtual model corresponding to the real equipment, real-time monitoring of equipment status, fault prediction and management can be achieved. The following is a detailed analysis of the background technology and the technical problems to be solved.

[0003] In the process of implementing digital twin technology, multiple servo systems are involved, including dynamic solution servo, force feedback output servo, etc. In practical applications, different servos in the system have different update frequencies. For example, dynamic solution may require a higher update rate to ensure real-time response, while force feedback output may have a lower update frequency due to computational complexity. This inconsistency can cause delayed or inaccurate system response, which in turn affects user experience and safety. During the simulation process, the update rate of each servo will be affected by many factors (such as computing load, network latency, etc.), causing its update frequency to fluctuate. This will make the feedback of some real-time interactive operations not smooth enough, reducing the overall performance of the system.

[0004] Therefore, how to effectively handle the coupling and synchronization problems between multi-update rate servos during the digital twin interaction process, provide stable and efficient support for the digital twin application of marine electromechanical equipment, and thus improve management efficiency and safety, is an urgent problem to be solved. Summary of the invention

[0005] The present application provides a digital twin interaction method, device, equipment and medium, which solves the technical problem of how to effectively handle the coupling and synchronization between multi-update rate servos during the digital twin interaction process, achieves the coupling and synchronization between servos, and improves the technical effect of the continuity of the interaction effect.

[0006] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0007] In a first aspect, an embodiment of the present application provides a digital twin interaction method, which is applied to a digital twin interaction system, wherein the system includes a tactile device, a virtual object, a dynamics solution servo, and a force feedback output servo; the method includes:

[0008] Acquire a position change amount of the tactile device interacting with the virtual object in a sampling period;

[0009] generating an estimated position of the tactile device based on the position change, the lag time of the dynamics solution servo, and the smoothing time of the force feedback output servo;

[0010] determining an estimated feedback force corresponding to the estimated position;

[0011] Performing error judgment on the estimated feedback force, and determining a transition feedback force according to the judgment result;

[0012] The transition feedback force is converted according to a preset mapping coefficient to obtain a target feedback force of the virtual object on the haptic device.

[0013] A digital twin interaction method provided in this embodiment can understand the user's operation dynamics in real time by obtaining the position change during the interaction between the tactile device and the virtual object. On this basis, the estimated position of the tactile device is calculated in combination with the lag time and smoothing time, thereby ensuring smoother interaction and reducing response delays. Next, the corresponding estimated feedback force is determined based on the estimated position to ensure the user's perception consistency in the virtual environment. By making an error judgment on the estimated feedback force, the feedback force is automatically adjusted to improve the accuracy of tactile feedback and avoid force feedback delay or dislocation. Finally, the transition feedback force is converted into a target feedback force, and accurate feedback on the virtual object is achieved through a preset mapping coefficient, thereby enhancing the user's sense of immersion and participation. The whole process achieves efficient and accurate force feedback, improving the usability of virtual reality or augmented reality applications.

[0014] Optionally, the acquisition of the position change includes the starting position of the tactile device; and the generating of the estimated position of the tactile device based on the position change, the lag time of the dynamic solution servo, and the smoothing time of the force feedback output servo includes:

[0015] Determining a total error time according to a lag time of the dynamics solution servo and a smoothing time of the force feedback output servo;

[0016] The estimated position is determined according to the starting position, the total error time and the position change.

[0017] This embodiment determines the total error time by calculating the lag time of the dynamic solution servo and the smoothing time of the force feedback output servo, thereby improving the ability to compensate for signal transmission and processing delays and enhancing response accuracy. Then, the estimated position is calculated using the starting position, total error time, and position change, which can predict the actual position of the tactile device in real time. This prediction capability ensures timely adjustment of the feedback force, allowing users to experience more consistent and natural feedback when interacting with virtual objects.

[0018] Optionally, determining the estimated feedback force corresponding to the estimated position includes:

[0019] constructing a rendering model for interaction between the haptic device and the virtual object using a generalized-damping-mass model;

[0020] An estimated feedback force corresponding to the estimated position is determined according to the drawing model.

[0021] This embodiment uses the generalized-damping-mass model to construct a drawing model for the interaction between a tactile device and a virtual object. The application of this model can simulate physical phenomena more realistically, thereby providing a more accurate interactive experience. Through this model, multiple factors, such as the interaction between damping and masses, can be taken into account, which improves the understanding of the dynamic characteristics of virtual objects. Then, the estimated feedback force corresponding to the estimated position is determined according to the drawing model, ensuring that the feedback force is closely related to the actual situation of the user's operation, so that the user can experience more natural and consistent tactile feedback during the interaction process.

[0022] Optionally, performing error judgment on the estimated feedback force and determining the transition feedback force according to the judgment result includes:

[0023] Acquire a hysteresis actual position of the tactile device at the hysteresis time, and a hysteresis estimated position of the tactile device obtained based on the position change and the hysteresis time;

[0024] Determining a position error rate corresponding to the haptic device based on the hysteresis estimated position and the hysteresis actual position;

[0025] When the position error rate is less than or equal to a preset error threshold, determining the estimated feedback force as the transition feedback force;

[0026] When the position error rate is greater than a preset error threshold, error correction is performed on the estimated feedback force to obtain the transition feedback force.

[0027] This embodiment ensures that the position error rate can be dynamically detected by real-time monitoring of the actual hysteresis position and the estimated hysteresis position of the tactile device. When the position error rate is less than or equal to the preset threshold, the estimated feedback force is directly output as the transition feedback force, thereby providing a smooth and natural user experience; when the error rate is greater than the threshold, error correction is performed to improve the accuracy of tactile feedback and significantly improve the user experience in virtual interaction.

[0028] Optionally, when the position error rate is greater than a preset error threshold, performing error correction on the estimated feedback force to obtain the transition feedback force includes:

[0029] Acquire a hysteresis estimated operation force of the haptic device at the hysteresis estimated position and a hysteresis actual operation force at the hysteresis actual position;

[0030] Performing a dynamics solution according to the hysteresis estimated operating force to obtain a hysteresis fitting feedback force, and performing a dynamics solution according to the hysteresis actual operating force to obtain a hysteresis actual feedback force;

[0031] Based on the hysteresis estimated operating force, the hysteresis actual operating force, the hysteresis fitting feedback force and the hysteresis actual feedback force, a feedback force error compensation amount is obtained by solving a preset dynamic model;

[0032] The estimated feedback force is corrected according to the feedback force error compensation amount to obtain the transitional feedback force.

[0033] This embodiment first obtains the operating force of the tactile device at the hysteresis estimated position and the hysteresis actual position. On this basis, the hysteresis fitting feedback force and the hysteresis actual feedback force are respectively obtained through dynamic solution to ensure accurate modeling of the device feedback state. Then, using the preset dynamic model, the feedback force error compensation amount is calculated according to some of the obtained operating forces and feedback forces. This compensation amount can effectively correct the estimated feedback force and finally obtain a more accurate transition feedback force. In summary, this process realizes the precise adjustment of tactile feedback, improves the user experience, and reduces the problem of tactile feedback delay or inconsistency caused by position error.

[0034] Optionally, the correcting the estimated feedback force according to the feedback force error compensation amount to obtain the transition feedback force includes:

[0035] generating a feedback force to be corrected according to the error compensation amount and the estimated feedback force;

[0036] A cubic parameter spline curve fitting is performed on the feedback force to be corrected to obtain the transition feedback force.

[0037] This embodiment generates the feedback force to be corrected by combining the error compensation amount and the estimated feedback force. Then, a cubic parameter spline curve is fitted to the feedback force to be corrected, which helps to smoothly transition the change of the feedback force and reduce the abruptness and unnatural feedback. Through this fitting method, smoother tactile feedback can be achieved, allowing users to feel a more natural and realistic experience during interaction.

[0038] Optionally, the kinetic model is:

[0039]

[0040]

[0041]

[0042] Among them, F 操作 is the estimated position of the tactile device at the lag p (K+1)T The delayed estimated operating force; F 输入 The tactile device is lagging behind the actual position The actual operating force of hysteresis; F δ is the operating force error compensation between the actual operating force and the estimated operating force; 预估 To estimate the operating force F based on the hysteresis 操作 The hysteresis fitting feedback force obtained by dynamic solution; f 反馈 is the lagging actual feedback force obtained by dynamics solution based on the lagging actual operating force; f δ is the feedback force error compensation between the delayed actual feedback force and the delayed estimated operating force; M is the mass matrix; B is the damping matrix; G is the stiffness matrix; q is the actual displacement of the virtual object, is the actual speed of the virtual object; is the actual acceleration of the virtual object; is the estimated displacement of the virtual object, is the estimated speed of the virtual object; is the estimated acceleration of the virtual object; K is the lag time; u is the smoothing time; T is the sampling period; t is the sampling time.

[0043] In a second aspect, an embodiment of the present application provides a digital twin interaction device, which is applied to a digital twin interaction system, wherein the system includes a tactile device, a virtual object, a dynamics solution servo, and a force feedback output servo; the device includes:

[0044] A position change acquisition module, used to acquire a position change amount of the tactile device interacting with the virtual object in a sampling period;

[0045] An estimated position generating module, configured to generate an estimated position of the tactile device based on the position variation, the lag time of the dynamics solution servo, and the smoothing time of the force feedback output servo;

[0046] An estimated feedback force determination module, used to determine the estimated feedback force corresponding to the estimated position;

[0047] An error judgment module, used for making an error judgment on the estimated feedback force and determining a transition feedback force according to the judgment result;

[0048] The target feedback force determination module is used to convert the transition feedback force according to a preset mapping coefficient to obtain a target feedback force of the virtual object on the haptic device.

[0049] In a third aspect, an embodiment of the present application provides a computer device, including:

[0050] A memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned digital twin interaction method by executing the computer instructions.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the digital twin interaction method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of a digital twin interaction method provided in an embodiment of the present application;

[0054] Figure 2 A drawing model provided for an embodiment of the present application;

[0055] Figure 3 A flowchart of step S3 provided in an embodiment of the present application;

[0056] Figure 4 A flowchart of step S5 provided in an embodiment of the present application;

[0057] Figure 5 A flowchart of step S7 provided in an embodiment of the present application;

[0058] Figure 6 A flowchart of step S77 provided in an embodiment of the present application;

[0059] Figure 7 A flowchart of step S777 provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of a synchronization method for a multi-update rate servo provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the rotation of a skeletal joint around a fixed point in three dimensions provided in an embodiment of the present application;

[0062] Fig.10 A schematic diagram of differentiating human skeleton joint chains provided in an embodiment of the present application;

[0063] Fig.11 A schematic diagram of a three-dimensional skeletal joint rotation analysis provided in an embodiment of the present application;

[0064] Fig.12 An interactive classification diagram of motion postures of equipment operations in a virtual environment provided by an embodiment of the present application;

[0065] Fig.13 A schematic diagram of a visualized interactive operation in a virtual environment provided by an embodiment of the present application;

[0066] Fig.14 A block diagram of a digital twin interaction device provided in an embodiment of the present application;

[0067] Fig.15 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0069] The application of digital twin technology in the health management of marine electromechanical equipment has become a key means to improve the efficiency and safety of equipment maintenance. By building a virtual model corresponding to the real equipment, that is, a virtual object, real-time monitoring of equipment status, fault prediction and management can be achieved. In the process of implementing digital twin technology, multiple servo systems are involved, including dynamic solution servo, force feedback output servo, etc., and the following technical problems exist:

[0070] Coupling and synchronization of multiple update rates: In practical applications, different servos in the system have different update frequencies. For example, dynamics solving may require a higher update rate to ensure real-time response, while force feedback output may have a lower update frequency due to computational complexity. This inconsistency can cause delayed or inaccurate system response, which in turn affects user experience and safety.

[0071] Fluctuation problem: During the simulation process, the update rate of each servo will be affected by many factors (such as computing load, network delay, etc.), causing its update frequency to fluctuate. This will make the feedback of some real-time interactive operations not smooth enough and reduce the overall performance of the system.

[0072] According to an embodiment of the present application, an embodiment of a digital twin interaction method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0073] In this embodiment, a digital twin interaction method is provided, which is applied to a digital twin interaction system. The system includes a tactile device, a virtual object, a dynamics solution servo, and a force feedback output servo. Figure 1 A flowchart of a digital twin interaction method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the process includes the following steps:

[0074] Step S1, obtaining a position change amount of the haptic device interacting with the virtual object in a sampling period.

[0075] Specifically, a haptic device is a device that can provide tactile feedback, which enables users to feel touch and force feedback in a virtual environment by simulating real tactile sensations, such as pressure, vibration, temperature, etc. A virtual object refers to an object or entity created in a virtual environment, which is a part model of a marine electromechanical equipment in the embodiment of the present application. The position, angle and other mechanical data of the haptic device during the interaction between the haptic device and the virtual object are detected and acquired by the sensor, and the position change of the haptic device can be calculated through these data.

[0076] Step S3, generating an estimated position of the tactile device based on the position change, the lag time of the dynamic solution servo, and the smoothing time of the force feedback output servo.

[0077] Since the update frequencies of the dynamic solution servo and the force feedback output servo may be inconsistent and fluctuate during the interactive simulation between the tactile device and the virtual object, the dynamic solution servo and the force feedback output servo need to be coupled and synchronized.

[0078] Specifically, the lag time of the dynamic solution servo refers to the time required from issuing a control command to responding. This time delay will affect the real-time response of the tactile device to the virtual object. The smoothing time of the force feedback output servo refers to the time required for the force change from the beginning to the stability when the tactile device outputs force feedback. This smoothing time needs to be taken into account to ensure the smoothness of the tactile feedback. In the period T, the position of the tactile device changes from p to 0 Change to p t , the position change is ; Considering the lag time KT of the dynamic solution servo and the smoothing time uT of the force feedback output servo, at the time (K+u+1)T, the estimated position of the tactile device is: .

[0079] Step S5, determining the estimated feedback force corresponding to the estimated position.

[0080] Use the generalized-damped-mass model to build a rendering model for haptic devices to interact with virtual objects, see Figure 2 :

[0081]

[0082]

[0083]

[0084] in, To draw the linear displacement of the model; is the angular displacement of the plotted model; s is the linear displacement mapping coefficient between the tactile device and the drawing model; r is the angular displacement mapping coefficient between the tactile device and the drawing model; x 1 is the current position of the tactile device; x 0 is the position of the tactile device at the last moment; θ 1 is the current angle of the tactile device; θ 0 K is the angle of the tactile device at the last moment; s , K t is the generalized stiffness matrix; C s is the velocity matrix; b r is the generalized damping matrix; is the linear speed of the drawing model; is the angular velocity of the drawn model; F s is the operating force; z is the generalized mass matrix of the virtual object; is the acceleration; μ is the Lagrange multiplier; Φ is the constraint equation of the virtual object; F t is the feedback force; m is the generalized mass matrix of the virtual object; is the speed of the virtual object; q i is the displacement of the virtual object.

[0085] Specifically, the estimated feedback force can be obtained by using the drawing model according to the estimated position. .

[0086] Step S7, performing error judgment on the estimated feedback force, and determining the transition feedback force according to the judgment result.

[0087] Specifically, error judgment of the estimated feedback force usually involves comparing the estimated feedback force with the actual feedback force to determine the size of the error. The error may be caused by a variety of factors, including the lag time of the dynamic solution servo, the smoothing time of the force feedback output servo, and the complexity of the interaction between the tactile device and the virtual object. Based on the result of the error judgment, the transition feedback force can be determined. The transition feedback force refers to the force that smoothly transitions between the current estimated feedback force and the final feedback force. This transition process is achieved by using methods such as interpolation. The determination of the transition feedback force aims to reduce sudden changes caused by errors, thereby providing a more natural and coherent tactile interaction experience.

[0088] Step S9: converting the transition feedback force according to a preset mapping coefficient to obtain a target feedback force of the virtual object on the haptic device.

[0089] Specifically, the mapping coefficient is a pre-set parameter used to convert the feedback force of the haptic device into a force suitable for user perception. For different virtual environments, the mapping coefficient is different, and the value of the mapping coefficient is set according to the characteristics of the virtual object, or provided by the supplier of the virtual object and the haptic device. According to the obtained transition feedback force, it is multiplied by the corresponding mapping coefficient to obtain the target feedback force. Finally, the target feedback force is output to the haptic device to generate tactile feedback that the user can perceive.

[0090] A digital twin interaction method provided in this embodiment can understand the user's operation dynamics in real time by obtaining the position change during the interaction between the tactile device and the virtual object. On this basis, the estimated position of the tactile device is calculated in combination with the lag time and smoothing time, thereby ensuring smoother interaction and reducing response delays. Next, the corresponding estimated feedback force is determined based on the estimated position to ensure the user's perception consistency in the virtual environment. By making an error judgment on the estimated feedback force, the feedback force is automatically adjusted to improve the accuracy of tactile feedback and avoid force feedback delay or dislocation. Finally, the transition feedback force is converted into a target feedback force, and accurate feedback on the virtual object is achieved through a preset mapping coefficient, thereby enhancing the user's sense of immersion and participation. The whole process achieves efficient and accurate force feedback, improving the usability of virtual reality or augmented reality applications.

[0091] Figure 3 In the flowchart of step S3 provided in the embodiment of the present application, the acquisition of the position change includes the starting position of the tactile device; the process may include the following steps:

[0092] Step S31, determining the total error time according to the lag time of the dynamic solution servo and the smoothing time of the force feedback output servo.

[0093] Specifically, within a sampling period T, the lag time is KT, the smoothing time is uT, and the total error time is the sum of the lag time and the smoothing time, plus an additional sampling period T to account for the overall delay within the current sampling period. Therefore, the total error time is (K+u+1)T.

[0094] Step S33, determining the estimated position according to the starting position, the total error time and the position change.

[0095] Specifically, within the period T, the position of the tactile device changes from the starting position p 0 Change to the final position p t , the position change is ; Considering the lag time KT of the dynamic solution servo and the smoothing time uT of the force feedback output servo, at the time (K+u+1)T, the estimated position of the tactile device is: .

[0096] and Figure 1 Compared with the embodiment shown in FIG. 1 , this embodiment determines the total error time by calculating the lag time of the dynamic solution servo and the smoothing time of the force feedback output servo, thereby improving the compensation capability for signal transmission and processing delays and enhancing the response accuracy. Then, the estimated position is calculated using the starting position, the total error time, and the position change, so that the actual position of the tactile device can be predicted in real time. This prediction capability ensures timely adjustment of the feedback force, allowing the user to experience more consistent and natural feedback when interacting with virtual objects.

[0097] Figure 4 The flowchart of step S5 provided in the embodiment of the present application may include the following steps:

[0098] Step S51 : constructing a rendering model for the interaction between the haptic device and the virtual object using a generalized-damping-mass model.

[0099] Draw the model as:

[0100]

[0101]

[0102]

[0103] in, To draw the linear displacement of the model; is the angular displacement of the plotted model; s is the linear displacement mapping coefficient between the tactile device and the drawing model; r is the angular displacement mapping coefficient between the tactile device and the drawing model; x 1 is the current position of the tactile device; x 0is the position of the tactile device at the last moment; θ 1 is the current angle of the tactile device; θ 0 K is the angle of the tactile device at the last moment; s , K t is the generalized stiffness matrix; C s is the velocity matrix; b r is the generalized damping matrix; is the linear speed of the drawing model; is the angular velocity of the drawn model; F s is the operating force; z is the generalized mass matrix of the virtual object; is the acceleration; μ is the Lagrange multiplier; Φ is the constraint equation of the virtual object; F t is the feedback force; m is the generalized mass matrix of the virtual object; is the speed of the virtual object; q i is the displacement of the virtual object.

[0104] Step S53: determining an estimated feedback force corresponding to the estimated position according to the drawn model.

[0105] Specifically, the estimated feedback force can be obtained by using the drawing model according to the estimated position. .

[0106] and Figure 1 Compared with the embodiment shown in the figure, this embodiment uses a generalized-damping-mass model to construct a drawing model for the interaction between a tactile device and a virtual object. The application of this model can simulate physical phenomena more realistically, thereby providing a more accurate interactive experience. Through this model, multiple factors can be taken into account, such as the interaction between damping and masses, which improves the understanding of the dynamic characteristics of virtual objects. Then, the estimated feedback force corresponding to the estimated position is determined according to the drawing model, ensuring that the feedback force is closely related to the actual situation of the user's operation, so that the user can experience more natural and consistent tactile feedback during the interaction process.

[0107] Figure 5 The flowchart of step S7 provided in the embodiment of the present application may include the following steps:

[0108] Step S71 , obtaining the actual hysteresis position of the haptic device at the hysteresis time, and the estimated hysteresis position of the haptic device obtained based on the position change and the hysteresis time.

[0109] Specifically, get the time t 0 +(K+1)T the actual position of the tactile device In the period T, the position of the tactile device changes from the starting position p 0 Change to the final position p t , the position change is ; According to the position change And the lag time KT, calculated at time t 0 +(K+1)T delayed estimated position .

[0110] Step S73 : determining a position error rate corresponding to the haptic device based on the hysteresis estimated position and the hysteresis actual position.

[0111] Specifically, the difference between the delayed estimated position and the delayed actual position is calculated ;

[0112] Calculate the time t based on the difference 0 Position error rate of the tactile device at +(K+u+1)T .

[0113] Step S75: when the position error rate is less than or equal to the preset error threshold, the estimated feedback force is determined as the transition feedback force.

[0114] Specifically, the position error rate δ is an indicator to measure the difference between the estimated position and the actual position. When the position error rate is less than or equal to the preset error threshold, it indicates that the estimated position of the tactile device is close enough to the actual position, so the estimated feedback force can be considered accurate and can be used as a transition feedback force. The preset error threshold here is preferably 5%.

[0115] Step S77, when the position error rate is greater than a preset error threshold, error correction is performed on the estimated feedback force to obtain a transition feedback force.

[0116] Specifically, when the position error rate is greater than the preset error threshold, it indicates that there is a large deviation between the estimated position and the actual position of the tactile device, and the estimated feedback force needs to be corrected to ensure the accuracy of the tactile interaction. The least squares method can be used for interpolation fitting to obtain the discrete values ​​at each moment, and the transition feedback force is output to the tactile device.

[0117] and Figure 1 Compared with the embodiment shown in the figure, this embodiment ensures that the position error rate can be dynamically detected by real-time monitoring of the actual hysteresis position and the estimated hysteresis position of the tactile device. When the position error rate is less than or equal to the preset threshold, the estimated feedback force is directly output as the transition feedback force, thereby providing a smooth and natural user experience; and when the error rate is greater than the threshold, error correction is performed to improve the accuracy of tactile feedback and significantly improve the user experience in virtual interaction.

[0118] Figure 6 The flowchart of step S77 provided in the embodiment of the present application may include the following steps:

[0119] Step S771 , obtaining the hysteresis estimated operation force of the haptic device at the hysteresis estimated position and the hysteresis actual operation force at the hysteresis actual position.

[0120] Specifically, according to the lag estimated position, by drawing Solve to get the corresponding hysteresis estimated operating force; according to the actual hysteresis position, by drawing the The corresponding hysteresis actual operating force is obtained by solving the problem.

[0121] Step S773, performing a dynamic solution based on the hysteresis estimated operating force to obtain a hysteresis fitting feedback force, and performing a dynamic solution based on the hysteresis actual operating force to obtain a hysteresis actual feedback force.

[0122] Specifically, the corresponding hysteresis fitting feedback force and hysteresis actual feedback force can be obtained by drawing the model solution.

[0123] Step S775, based on the hysteresis estimated operating force, the hysteresis actual operating force, the hysteresis fitting feedback force and the hysteresis actual feedback force, a preset dynamic model is used to solve and obtain a feedback force error compensation amount.

[0124] Specifically, the preset kinetic model is used to solve t 0 +(K+u+1)T time feedback force error compensation .

[0125] In some embodiments, the kinetic model is:

[0126]

[0127]

[0128]

[0129] Among them, F 操作 is the estimated position of the tactile device at the lag p (K+1)T The delayed estimated operating force; F 输入 The tactile device is lagging behind the actual position The actual operating force of hysteresis; F δ is the operating force error compensation between the actual operating force and the estimated operating force; 预估 To estimate the operating force F based on the hysteresis 操作 The hysteresis fitting feedback force obtained by dynamic solution; f 反馈 is the lagging actual feedback force obtained by dynamics solution based on the lagging actual operating force; f δ is the feedback force error compensation between the delayed actual feedback force and the delayed estimated operating force; M is the mass matrix; B is the damping matrix; G is the stiffness matrix; q is the actual displacement of the virtual object, is the actual speed of the virtual object; is the actual acceleration of the virtual object; is the estimated displacement of the virtual object, is the estimated speed of the virtual object; is the estimated acceleration of the virtual object; K is the lag time; u is the smoothing time; T is the sampling period; t is the sampling time.

[0130] Step S777, correcting the estimated feedback force according to the feedback force error compensation amount to obtain a transition feedback force.

[0131] Specifically, the feedback force to be corrected is generated according to the feedback force error compensation amount:

[0132]

[0133] Then the feedback force to be corrected is corrected by using interpolation and other methods at t 0 +(K+1)T to t 0 +(K+u+1)T generates transition feedback force at each moment .

[0134] and Figure 5 Compared with the embodiment shown in the figure, this embodiment first obtains the operating force of the tactile device at the hysteresis estimated position and the hysteresis actual position, and on this basis, obtains the hysteresis fitting feedback force and the hysteresis actual feedback force respectively through dynamic solution, so as to ensure the accurate modeling of the device feedback state. Then, using the preset dynamic model, the feedback force error compensation amount is calculated according to some of the obtained operating forces and feedback forces. This compensation amount can effectively correct the estimated feedback force and finally obtain a more accurate transition feedback force. In summary, this process realizes the precise adjustment of tactile feedback, improves the user experience, and reduces the problem of tactile feedback delay or inconsistency caused by position error.

[0135] Figure 7 The flowchart of step S777 provided in the embodiment of the present application may include the following steps:

[0136] Step S7771, generating the feedback force to be corrected according to the error compensation amount and the estimated feedback force.

[0137] Specifically, the feedback force to be corrected is:

[0138]

[0139] Step S7773, performing cubic parameter spline curve fitting on the feedback force to be corrected to obtain a transition feedback force.

[0140] Specifically, the curve parameter equation of the cubic parametric spline curve fitting is:

[0141]

[0142] Solve the parameters a, b, and c of the cubic spline curve according to the boundary conditions. 0 +(K+1)T to t 0 At each moment of +(K+u+1)T, the transition feedback force is obtained using the parametric equation The transition feedback force can then be The target feedback force of the virtual object on the tactile device is converted according to the preset mapping coefficient and input into the tactile device. 0 +(K+u+1)T Finally, the dynamic state parameters obtained at this moment are used as the initial values ​​of the next cycle of each servo coupling.

[0143] and Figure 6 Compared with the embodiment shown in FIG. 1 , this embodiment generates a feedback force to be corrected by combining the error compensation amount and the estimated feedback force. Then, a cubic parameter spline curve is fitted to the feedback force to be corrected, which helps to smoothly transition the change of the feedback force and reduce the abruptness and unnatural feedback. Through this fitting method, a smoother tactile feedback can be achieved, allowing users to feel a more natural and realistic experience during interaction.

[0144] Figure 8 A schematic diagram of a synchronization method for a multi-update rate servo provided in an embodiment of the present application is shown in FIG. Figure 8 As shown in the figure, as the interactive simulation runs, the number of component models involved in scene drawing and dynamics simulation may change continuously, and the rates of the dynamics solving servo and the graphics refreshing servo are not constant. Therefore, it is necessary to synchronize the servos with different update rates to ensure the consistency of the operator's visual and tactile experience.

[0145] 1. Graphics refresh servo: The number of polygonal patches in the scene is the key factor in determining the refresh frame rate. In each refresh cycle, the number of polygonal patches to be displayed is counted and compared with the data of the previous cycles:

[0146] If the number of polygons increases significantly: Use Level of Detail (LOD) to simplify the display of minor components to reduce the number of polygons and increase the refresh rate.

[0147] If the number of polygons is significantly reduced: increase the sleep time of the graphics servo thread proportionally to allocate more resources to the dynamics solver servo and force feedback servo.

[0148] Through dynamic adjustment, the interval between scene refreshes is kept within a stable range.

[0149] 2. Dynamic solution servo: The smaller the integral step size, the higher the accuracy of the solution. However, too small a step size will lead to too many iterations, affecting real-time performance. If the step size is too large, matrix singularity may occur, affecting calculation accuracy.

[0150] By calculating the load balance model, the dynamic integration step size is optimized to ensure the real-time calculation and the accuracy of the simulation results. After the solution is completed, the efficiency of the current simulation model is evaluated, and the number of integration iterations and the time required for calculation are recorded:

[0151] The efficiency is significantly reduced: appropriately increase the integration step size and submit a resource request signal to shorten the sleep time of the dynamics servo.

[0152] Improved efficiency: proportionally shorter integration steps for more precise force feedback output.

[0153] 3. Force feedback output servo: Force feedback servo is usually set to the highest priority, with a very short sleep time, consuming a lot of computing resources.

[0154] To meet the real-time requirements of the simulation, the load balance model dynamically adjusts the frequency of the force feedback servo. When the graphics refresh and dynamics solution servos submit resource request signals, the frequency of the force feedback servo is reduced according to the signal strength, but maintained at no less than 800 Hz to release resources for the other two servos.

[0155] Through the above method, the model drawing details of the graphics refresh servo are dynamically switched, the integral step of the dynamic solution servo is optimized, and the refresh frequency of the force feedback servo is adjusted in real time to meet the real-time requirements of dynamic interactive simulation.

[0156] It should be noted that when the scene graphics are refreshed, the positions of the virtual objects, i.e., the components, need to be updated according to the results of the dynamics solution. At the same time, the force feedback servo outputs the corresponding feedback force, so that the operator can feel the force required to drag the part model from the previous position state to the current state. To ensure that the graphics display and force feedback output are consistent in vision and touch, the dynamics solution cannot be refreshed across the previous and next scenes. Therefore, when the scene refresh is completed, the servos need to be synchronized. The specific steps are as follows:

[0157] The dynamic adjustment method ensures that graphics refreshing, dynamics solving and force feedback servo are completed at the same time.

[0158] The feedback force data obtained by the dynamics solving servo is transmitted to the force feedback servo, and the position and posture data of the component are transmitted to the graphics refreshing servo.

[0159] After waiting for the graphics refresh to complete, restart the graphics refresh, dynamics calculation and force feedback servo to perform the next cycle of interactive simulation.

[0160] In order to achieve the synchronization between dynamic solution and graphics refresh servo, the graphics refresh cycle (JT) obtained by calculating the load balancing model is adjusted by adjusting the motion trend estimation time (k n ) and feedback force balance transition time μ n , so that the time of the two is roughly the same, as shown in the following formula: nT+k 1 T+μ 1 T+…+k n T+μ n T≤JT. Since the frequency of the force feedback servo is much higher than the scene refresh frequency, synchronization can be achieved by simply keeping the feedback force value constant for a few steps before the refresh.

[0161] The embodiment of the present application also provides equipment virtual reality interaction technology based on human skeleton posture recognition. According to the needs of equipment operation process simulation in a virtual environment, the naturalness and friendliness of interaction are particularly important. In order to make the virtual simulation closer to the equipment operation process in reality, virtual humans are introduced as interactive media. These virtual humans are given richer interactive operations through the interactive semantic expressions of real people, thereby achieving a more comprehensive and realistic user experience.

[0162] Specifically, the virtual human skeleton mapping and multi-chain joint kinematic model are established:

[0163] In the three-dimensional environment of equipment operation, in order to make the operation simulation process closer to the real human operation, a virtual human model with multiple degrees of freedom is added. The interactive operation of the virtual human model is processed and analyzed through the bone data and color and depth image data collected by Kinect to realize the mapping of joint binding drive and corresponding instruction mode. The bone information provided by Kinect contains 25 joint points, while the virtual human model has 55 joints. These joints include the main trunk, 4 each of the left and right arms and legs, as well as 5 head and 15 joint drive points each of the left and right hands. Considering the instability of Kinect's estimation of the thumb and fingertip joints, it was finally chosen to map the 16 joint points of the main trunk of the human body one by one. As shown in Table 1:

[0164] Table 1 Mapping table of skeleton nodes and Unity3d virtual human joints

[0165]

[0166] Define a bone dataset C=[c 1 ,c 2 ,c n-1 ,c n], using this data set as input to drive the movement of the virtual human body and establish a kinematic model of the virtual human joints in the digital twin environment. The joints of the virtual human body rotate around their joint points in space. Set a set of independent Euler angle parameters for each rotation joint . See Fig. 9 , from position A 0 Rotate to position A 1 The transformation can be expressed as: , where R, A 1 , N, A 0 are rotation operators, respectively:

[0167]

[0168]

[0169]

[0170] By multiplying the matrices, we can get the final rotation matrix:

[0171]

[0172] Among them, A 0 The point coordinates are (x 0 ,y 0 ,z 0 ), A 1 The point coordinates are (x 1 ,y 1 ,z 1 ), its coordinate transformation relationship is:

[0173]

[0174] According to the structure of the human skeleton, the movement can be divided into the movement of the main trunk, left and right arms, and left and right legs, which is specifically the movement of 5 joint chains. Figure 10-11 For each joint chain rotation, it is not a simple point rotation around a fixed point, but can be abstracted as the rotation of a skeleton chain connected by multiple joints. For example, the skeleton node appears as a green skeleton chain OA before rotation. 0 B 0 C 0 , after rotation, it appears as a pink bone chain OA 1 B 1 C 1 The required transformation can be expressed as: R = R A ·R B ·R C .

[0175] Through the formula, we can get:

[0176]

[0177] The above formula represents A 0 Around fixed point Rotate to A 1 , Around point A 1 Rotate to B 1 , Around point B 1 Rotate to C 1 , here It is B 0 Follow A 0 The new position obtained by rotation, The first time following A 0 Rotate, then follow a second time The new position obtained by rotation is finally C 1 .

[0178] Virtual human motion posture mapping and motion library establishment in equipment operation:

[0179] In the equipment operation process in the virtual environment, the operator usually takes certain basic fixed postures. These postures are indispensable in the equipment operation simulation interaction, and they represent specific interaction semantics. The purpose of the operator performing these interaction actions is to promote the equipment operation process or adjust the equipment operation parameters. At the same time, after receiving the interaction semantics, the virtual human will provide corresponding virtual interaction feedback.

[0180] According to different interaction purposes, the action gestures in the virtual device operation process are divided into two categories, see Fig.12 .

[0181] Mobile action posture: refers to the operator moving forward, backward, left, and right. Since the skeleton information directly drives the virtual human, there is no need to distinguish the mobile action posture, which is presented as the virtual human following the operator's movement changes in real time.

[0182] Operational gestures: refers to the actions of operators to operate accessories required for process operation during equipment operation. This type of interaction is mainly used to change the spatial position of accessories or materials in the equipment, or to promote the operation process of the equipment.

[0183] Based on the classification and definition of the above interactive action postures, an interactive action posture library for equipment operation in a virtual environment is established. The library includes the following contents for each action posture: action label, action name, action essentials, and action diagram. The constructed action posture library is convenient for training and recognition using the human action skeleton posture recognition algorithm, so as to calculate the interactive semantic category and trigger the corresponding posture action feedback of the virtual person. Table 2 lists the library of some operational action postures. It should be noted that since the mobile posture follows the operator's walking action in real time and stably, no special description is made in the table library.

[0184] Table 2 Simplified analysis of the five postures and their basic principles

[0185]

[0186] The following describes the visualization interaction of CAE data in the digital twin simulation of the present invention in conjunction with a specific application scenario.

[0187] In the digital twin simulation environment, parts can be grasped by a virtual hand equipped with a force feedback glove and have different constraint states. These constraints are divided into three cases: no constraints in the free state, partial degrees of freedom constraints (along the directed curves that constrain the movement of the part), and full degrees of freedom constraints (locating the position on the device).

[0188] 1. Partial freedom constraints

[0189] a. When the part is not grasped: the part is allowed to move autonomously by inertia. The linear and angular displacements of the part are checked in each frame. If the part deviates from the constraint curve due to linear displacement, it is repositioned to the projection point on the constraint curve and its velocity is preserved. The angular position of the part is always consistent with the tangent direction of the constraint curve.

[0190] b. When the part is grasped: the position of the part is determined by the virtual hand and remains relatively still. The position of the virtual hand is determined by the VR tracker, which remains free.

[0191] Specific steps:

[0192] Step 1: Record the relative pose of the hand to the VR tracker: Vector: 、Quaternion: .

[0193] Step 2: When the hand grabs the part, record the relative position of the hand to the part: Vector: 、Quaternion: .

[0194] Step 3: Constrain the part to the directed curve and update the new pose: vector and quaternions , keep the VR tracker free to move and get the new pose of the tracker: vector and quaternions .

[0195] Step 4: Calculate the new pose of the hand (if the hand grasps the part): Vector: 、Quaternion: .

[0196] Step 5: If you release the part, restore the original relative position: Vector: 、Quaternion: .

[0197] 2. Full freedom constraints

[0198] When a part is in a positioned position, it is allowed to be moved under certain conditions.

[0199] a. When the part is not grasped: the part maintains constraints in all degrees of freedom.

[0200] b. When the part is grasped: a certain speed limit must be exceeded to move the part. Check at each frame whether the displacement of the hand exceeds the threshold. If not, the movement is valid. When the part slides off the end of the constraint curve, it will enter the free state. If the part is in the free state and its axis point is close to the end of the constraint curve and the rotation direction is the same, it will be captured back to the positioning state.

[0201] See also Fig.13 , in the digital twin simulation scenario, the parts with time-varying stress fields are sectioned as follows:

[0202] Step 1001: In the digital twin simulation scene, the right hand uses a force feedback glove to pick up the part.

[0203] Step 1002: In the digital twin simulation scene, the left hand uses the handle of the virtual reality device to draw a cutting plane, and transforms the cutting plane to the coordinate space where the part model is located.

[0204] Step 1002-1: When the operation button of the handle is pressed, the cutting plane drawing state is entered. The frame at this time is called the initial frame, and the position and posture of the handle are recorded.

[0205] Step 1002-2: In the cutting plane drawing state, the front end of the handle keeps projecting a straight line indicator light of a fixed length in the space of the digital twin simulation scene, and updates the position of the light in each frame to keep it relatively still with the handle.

[0206] Step 1002-3: In the cutting plane drawing state, according to the position and posture of the handle in the initial frame and the position and posture of the handle in the current frame, the global coordinate system plane Q of the current frame is processed and visualized in a semi-transparent manner. By adjusting the posture of the part and the handle, the cutting plane can be placed at the desired position.

[0207] Step 1002-3-1: Set the starting point of the projection ray at the initial position of the handle as P 1 , the end point is located at P 2 ; Set the starting point of the projection ray of the handle's current frame position as P 3 , the end point is located at P 4 .

[0208] Step 1002-3-2: Take point P 1 , P 3 and P 4 The three points determine the plane Q, and then point P 2 The point projected onto plane Q is denoted as point .

[0209] Step 1002-3-3: Click P 1 ,point , click P 4 The triangles and points , click P 4 , click P 3 The resulting triangles have common edges and lie on plane Q. Use them to visualize plane Q.

[0210] Step 1002-4: When the operating button of the handle is released, the last global coordinate system plane is accepted as the sectioning plane, and the sectioning plane drawing state is exited.

[0211] Step 1003: pass the part model with time-varying field data and the transformed cutting plane parameters to the cutting processing module, and use the ClipDataSet filter of VTK to perform cutting. The specific steps are as follows:

[0212] Step 1003-1: Read in the part model with time-varying field data and generate vtkUnstructuredGrid type data of multiple time steps.

[0213] Step 1003 - 2 : According to the requirement on the number of key frames, the unstructured grid type data of multiple time steps are re-sampled uniformly to obtain the data of each key frame.

[0214] Step 1003-3: Use ClipDataSet filter to slice all key frames respectively to obtain vtkUnstructuredGrid type data after all key frames are sliced.

[0215] Step 1004: extract the surface and surface data of the part model after sectioning, and pass it back to the digital twin simulation scene to update the model to be sectioned. The specific operations are as follows:

[0216] The vtkGeometry filter and the vtkTriangleFilter filter are used to extract the geometry of the vtkUnstructuredGrid type data and the outer surface of the field data of each model part after sectioning, and generate vtkPolyData type data. After conversion, the triangle mesh and vertex data supported by the visualization scene are generated to update the outer surface of the original part model in the visualization scene.

[0217] Step 1005: Multiple sectioning planes may be added successively. For the sectioning result of the last part model, the above steps 1002 to 1003 are repeated to perform serial processing on the sectioned data.

[0218] Accordingly, please refer to Fig.14 A block diagram of a digital twin interaction device provided in an embodiment of the present application is applied to a digital twin interaction system. The system includes a tactile device, a virtual object, a dynamic solution servo, and a force feedback output servo. The device includes:

[0219] The position change acquisition module S101 is used to acquire the position change amount of the tactile device interacting with the virtual object in a sampling period;

[0220] An estimated position generation module S103 is used to generate an estimated position of the tactile device based on the position change, the lag time of the dynamic solution servo, and the smoothing time of the force feedback output servo;

[0221] An estimated feedback force determination module S105 is used to determine an estimated feedback force corresponding to the estimated position;

[0222] The error judgment module S107 is used to make an error judgment on the estimated feedback force and determine the transition feedback force according to the judgment result;

[0223] The target feedback force determination module S109 is used to convert the transition feedback force according to a preset mapping coefficient to obtain a target feedback force of the virtual object on the haptic device.

[0224] In some optional implementations, the acquisition of the position change includes the starting position of the tactile device; the estimated position generation module S103 includes:

[0225] The total error time is determined based on the lag time of the dynamic solution servo and the smoothing time of the force feedback output servo;

[0226] Determine the estimated position based on the starting position, total error time and position change.

[0227] In some optional implementations, the estimated feedback force determination module S105 includes:

[0228] A rendering model for the interaction between haptic devices and virtual objects is constructed using the generalized-damped-mass model;

[0229] The estimated feedback force corresponding to the estimated position is determined based on the drawn model.

[0230] In some optional implementations, the error determination module S107 includes:

[0231] Acquire the actual hysteresis position of the tactile device at the hysteresis time, and the estimated hysteresis position of the tactile device based on the position change and the hysteresis time;

[0232] Determining a position error rate corresponding to the tactile device based on the hysteresis estimated position and the hysteresis actual position;

[0233] When the position error rate is less than or equal to a preset error threshold, the estimated feedback force is determined as a transition feedback force;

[0234] When the position error rate is greater than a preset error threshold, the estimated feedback force is error corrected to obtain a transition feedback force.

[0235] In some optional implementations, when the position error rate is greater than a preset error threshold, error correction is performed on the estimated feedback force to obtain a transition feedback force, including:

[0236] Obtaining a hysteresis estimated operating force of the tactile device at a hysteresis estimated position and a hysteresis actual operating force at a hysteresis actual position;

[0237] Performing a dynamics solution based on the hysteresis estimated operating force to obtain a hysteresis fitting feedback force, and performing a dynamics solution based on the hysteresis actual operating force to obtain a hysteresis actual feedback force;

[0238] Based on the hysteresis estimated operating force, the hysteresis actual operating force, the hysteresis fitting feedback force and the hysteresis actual feedback force, the feedback force error compensation amount is obtained by using a preset dynamic model;

[0239] The estimated feedback force is corrected according to the feedback force error compensation amount to obtain the transition feedback force.

[0240] In some optional implementations, the estimated feedback force is corrected according to the feedback force error compensation amount to obtain the transition feedback force, including:

[0241] Generate a feedback force to be corrected according to the error compensation amount and the estimated feedback force;

[0242] The feedback force to be corrected is fitted with a cubic parameter spline curve to obtain the transition feedback force.

[0243] In some optional embodiments, the kinetic model is:

[0244]

[0245]

[0246]

[0247] Among them, F 操作 is the estimated position of the tactile device at the lag p (K+1)T The delayed estimated operating force; F 输入 The tactile device is lagging behind the actual position The actual operating force of hysteresis; F δ is the operating force error compensation between the actual operating force and the estimated operating force; 预估 To estimate the operating force F based on the hysteresis 操作 The hysteresis fitting feedback force obtained by dynamic solution; f 反馈 is the lagging actual feedback force obtained by dynamics solution based on the lagging actual operating force; f δ is the feedback force error compensation between the delayed actual feedback force and the delayed estimated operating force; M is the mass matrix; B is the damping matrix; G is the stiffness matrix; q is the actual displacement of the virtual object, is the actual speed of the virtual object; is the actual acceleration of the virtual object; is the estimated displacement of the virtual object, is the estimated speed of the virtual object; is the estimated acceleration of the virtual object; K is the lag time; u is the smoothing time; T is the sampling period; t is the sampling time.

[0248] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0249] A digital twin interaction device in this embodiment is presented in the form of a functional module, where the module refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0250] See also Fig.15 , Fig.15 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Fig.15As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.15 A processor 10 is taken as an example.

[0251] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0252] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0253] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0254] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0255] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0256] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0257] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0258] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0259] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0260] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0261] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0262] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0263] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0264] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0265] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0266] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A digital twin interaction method, characterized in that: Applied to a digital twin interaction system, the system includes a tactile device, a virtual object, a dynamics solution servo, and a force feedback output servo; the method includes: Acquire a position change amount of the tactile device interacting with the virtual object in a sampling period; generating an estimated position of the tactile device based on the position change, the lag time of the dynamics solution servo, and the smoothing time of the force feedback output servo; determining an estimated feedback force corresponding to the estimated position; Performing error judgment on the estimated feedback force, and determining a transition feedback force according to the judgment result; The step of performing error judgment on the estimated feedback force and determining the transition feedback force according to the judgment result includes: Acquire a hysteresis actual position of the tactile device at the hysteresis time, and a hysteresis estimated position of the tactile device obtained based on the position change and the hysteresis time; Determining a position error rate corresponding to the haptic device based on the hysteresis estimated position and the hysteresis actual position; When the position error rate is less than or equal to a preset error threshold, determining the estimated feedback force as the transition feedback force; When the position error rate is greater than a preset error threshold, performing error correction on the estimated feedback force to obtain the transition feedback force; Wherein, when the position error rate is greater than a preset error threshold, performing error correction on the estimated feedback force to obtain the transition feedback force includes: Acquire a hysteresis estimated operation force of the haptic device at the hysteresis estimated position and a hysteresis actual operation force at the hysteresis actual position; Performing a dynamics solution according to the hysteresis estimated operating force to obtain a hysteresis fitting feedback force, and performing a dynamics solution according to the hysteresis actual operating force to obtain a hysteresis actual feedback force; Based on the hysteresis estimated operating force, the hysteresis actual operating force, the hysteresis fitting feedback force and the hysteresis actual feedback force, a feedback force error compensation amount is obtained by solving a preset dynamic model; Correcting the estimated feedback force according to the feedback force error compensation amount to obtain the transition feedback force; The transition feedback force is converted according to a preset mapping coefficient to obtain a target feedback force of the virtual object on the haptic device.

2. The method according to claim 1, characterized in that The acquisition of the position change includes the starting position of the tactile device; the generating of the estimated position of the tactile device based on the position change, the lag time of the dynamic solution servo and the smoothing time of the force feedback output servo includes: Determining a total error time according to a lag time of the dynamics solution servo and a smoothing time of the force feedback output servo; The estimated position is determined according to the starting position, the total error time and the position change.

3. The method according to claim 1, characterized in that The determining the estimated feedback force corresponding to the estimated position includes: constructing a rendering model for interaction between the haptic device and the virtual object using a generalized-damping-mass model; An estimated feedback force corresponding to the estimated position is determined according to the drawing model.

4. The method according to claim 1, characterized in that: The step of correcting the estimated feedback force according to the feedback force error compensation amount to obtain the transitional feedback force includes: generating a feedback force to be corrected according to the error compensation amount and the estimated feedback force; A cubic parameter spline curve fitting is performed on the feedback force to be corrected to obtain the transition feedback force.

5. The method according to claim 1, characterized in that The kinetic model is: Among them, F 操作 is the estimated position of the tactile device at the lag p (K+1)T The delayed estimated operating force; F 输入 The tactile device is lagging behind the actual position The actual operating force of hysteresis; F δ is the operating force error compensation between the actual operating force and the estimated operating force; 预估 To estimate the operating force F based on the hysteresis 操作 The hysteresis fitting feedback force obtained by dynamic solution; f 反馈 is the lagging actual feedback force obtained by dynamics solution based on the lagging actual operating force; f δ is the feedback force error compensation between the delayed actual feedback force and the delayed estimated operating force; M is the mass matrix; B is the damping matrix; G is the stiffness matrix; q is the actual displacement of the virtual object, is the actual speed of the virtual object; is the actual acceleration of the virtual object; is the estimated displacement of the virtual object, is the estimated speed of the virtual object; is the estimated acceleration of the virtual object; K is the lag time; u is the smoothing time; T is the sampling period; t is the sampling time.

6. A digital twin interaction device, characterized in that: Applied to a digital twin interaction system, the system includes a tactile device, a virtual object, a dynamics solving servo and a force feedback output servo; the device includes: A position change acquisition module, used to acquire a position change amount of the tactile device interacting with the virtual object in a sampling period; An estimated position generating module, configured to generate an estimated position of the tactile device based on the position variation, the lag time of the dynamics solution servo, and the smoothing time of the force feedback output servo; An estimated feedback force determination module, used to determine the estimated feedback force corresponding to the estimated position; An error judgment module, used for making an error judgment on the estimated feedback force and determining a transition feedback force according to the judgment result; The step of performing error judgment on the estimated feedback force and determining the transition feedback force according to the judgment result includes: Acquire a hysteresis actual position of the tactile device at the hysteresis time, and a hysteresis estimated position of the tactile device obtained based on the position change and the hysteresis time; Determining a position error rate corresponding to the haptic device based on the hysteresis estimated position and the hysteresis actual position; When the position error rate is less than or equal to a preset error threshold, determining the estimated feedback force as the transition feedback force; When the position error rate is greater than a preset error threshold, performing error correction on the estimated feedback force to obtain the transition feedback force; Wherein, when the position error rate is greater than a preset error threshold, performing error correction on the estimated feedback force to obtain the transition feedback force includes: Acquire a hysteresis estimated operation force of the haptic device at the hysteresis estimated position and a hysteresis actual operation force at the hysteresis actual position; Performing a dynamics solution according to the hysteresis estimated operating force to obtain a hysteresis fitting feedback force, and performing a dynamics solution according to the hysteresis actual operating force to obtain a hysteresis actual feedback force; Based on the hysteresis estimated operating force, the hysteresis actual operating force, the hysteresis fitting feedback force and the hysteresis actual feedback force, a feedback force error compensation amount is obtained by solving a preset dynamic model; Correcting the estimated feedback force according to the feedback force error compensation amount to obtain the transition feedback force; The target feedback force determination module is used to convert the transition feedback force according to a preset mapping coefficient to obtain a target feedback force of the virtual object on the haptic device.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the digital twin interaction method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the digital twin interaction method described in any one of claims 1 to 5.

Images