Dexterous hand digital twinning system based on virtual-real interaction and development method thereof

Abstract

The invention discloses a dexterous hand digital twinning system based on virtual-real interaction and a development method thereof, and the method comprises the following steps: creating a dexterous hand model through a digital modeling tool, and binding a skeleton structure and motion information; the back-end server is built by a Django framework, a visual front-end page is designed, and a 3D model display window and an embedded video acquisition window are integrated; designing a back-end interface and a database architecture; on the basis of hand actions captured by a video window, accurate action mapping is carried out on the 3D model, and real-time generation of motion control data is realized; and the front end performs visualization processing on the data in the database by using a chart library and updates the data such as the motion state of the model and the joint angle in real time. According to the dexterous hand digital twinning system based on virtual-real interaction and the development method thereof, high-fidelity mirroring of the digital twinning system is realized, and meanwhile, real-time and accurate interactive transmission of data in a virtual-real space can be ensured based on a communication mechanism.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a dexterous hand digital twin system based on virtual-reality interaction and a development method thereof. Background Art

[0002] With the rapid development of intelligent manufacturing and virtual reality technology, people's demand for high-precision and high-flexibility hand operations is growing. In many fields such as medical surgery simulation, remote control, and virtual reality entertainment, it is necessary to achieve accurate capture and real-time feedback of hand movements. However, traditional hand operation technology is often limited by factors such as equipment accuracy and environmental interference, making it difficult to achieve seamless interaction between hand movements and virtual environments. As an emerging technical means, digital twin technology provides a possibility to solve the above problems by mapping and synchronizing entities in the physical world with models in the virtual world. The dexterous hand digital twin system based on virtual-real interaction can capture hand motion data in real time and map it to a virtual model to achieve synchronous interaction between hand operations and virtual environments. This system can not only improve the accuracy and flexibility of operations, but also provide operators with a more intuitive and realistic operation experience. Therefore, the development of a dexterous hand digital twin system based on virtual-real interaction is of great significance to promoting the development of intelligent manufacturing, virtual reality and other fields, and is also an important way to meet people's needs for high-precision hand operations.

[0003] Although digital twin and visualization technologies have made significant progress in recent years, existing technologies still have significant defects in realizing virtual-reality interaction between hand movements and virtual environments. Traditional hand motion recognition technology is limited by equipment accuracy and environmental interference, making it difficult to achieve accurate capture, resulting in a large deviation between virtual model movements and actual hand operations, affecting operational accuracy and realism. At the same time, most existing digital twin systems lack real-time and interactivity, and can only statically display the state of physical entities, and cannot achieve real-time interaction with virtual environments, limiting their application in dynamic operation scenarios. In addition, the application of visualization technology in the interaction between hand movements and virtual environments is also insufficient. It is difficult to intuitively display the subtle differences and dynamic changes in hand movements, making it difficult for operators to accurately understand the motion state of virtual models. Summary of the invention

[0004] The purpose of the present invention is to provide a dexterous hand digital twin system based on virtual-reality interaction and a development method thereof, which fully utilizes the front-end interaction advantages of Vue and the three-dimensional rendering capabilities of Unity, and realizes high-fidelity mirroring of the digital twin system by constructing a front-end interaction interface, back-end processing logic, database management system and digital twin model; at the same time, by designing a set of efficient communication mechanisms, it ensures that data in the virtual and real space can be interactively transmitted in real time and accurately.

[0005] To achieve the above object, the present invention provides a method for developing a dexterous hand digital twin system based on virtual-reality interaction, comprising the following steps:

[0006] Step S1, using digital modeling tools to create a dexterous hand model, and binding the bone structure and motion information to it;

[0007] Step S2, designing a visualization front-end page, which integrates a 3D model display window for real-time presentation of the motion state of the dexterous hand; and embeds a video acquisition window to capture and analyze hand movements;

[0008] Step S3: The backend server is built with the Django framework, and the backend interface and database architecture are designed to save the key physical information of the dexterous hand and build an offline database;

[0009] Step S4: Based on the hand movements captured by the video window, image recognition and analysis technology is used to accurately map the movements of the 3D model, thereby achieving real-time generation of motion control data;

[0010] Step S5: design human-computer interaction to ensure data security and standardization during serial port transmission to prevent data leakage and tampering;

[0011] Step S6: The front end uses the chart library to visualize the data in the database and displays it on the page, while updating the model motion status and joint angle data in real time.

[0012] Preferably, in step S1, a dexterous hand model is constructed using a digital modeling tool, and the specific process is as follows:

[0013] Step S11, obtaining the joints and module size and shape information of the actual dexterous hand, and establishing the corresponding component models on the Unity platform; combining the component models, distinguishing between free joints and fixed joints, and assembling to form a complete virtual dexterous hand;

[0014] Step S12: Build a scene in Unity according to the environment information and place the component model into the scene; add texture maps to all components and scenes, add material balls, assign physical properties, and form geometric modeling;

[0015] Step S13, creating a C# script file, defining components and life cycle functions; adding action parent-child relationships to the 3D model to ensure that the virtual dexterous hand moves in accordance with the laws of physical motion;

[0016] Step S14: design the variables that need to be controlled for the model movement, define the speed and acceleration parameters; limit the movement range to ensure that the dexterous hand movements are within a safe range;

[0017] Step S15, add a complete action control function to realize the control of model movement by input control information; add sliding components, bind active joints, and directly control the model movement through components.

[0018] Preferably, in step S2, a visualization front-end page is designed, which integrates a 3D model display area and embeds a video acquisition window. The specific process is as follows:

[0019] Step S21, use IDEA development tool to create a Vue project and install project dependencies; introduce the Store module to implement global state management of the Vue project, and introduce the Element tool library as the front-end UI component library;

[0020] Step S22: introduce axios, configure asynchronous communication interceptor, build request framework; create request management js file, and globally manage all request configurations;

[0021] Step S23, create an application using the imported module and mount the Vue application;

[0022] Step S24, design the overall page and various components; introduce the Echarts drawing tool, and add a visual chart to display the dexterous hand model information;

[0023] Step S25, introduce the camera component, write the camera call method, and implement the camera integration window;

[0024] Step S26: Introduce the WebGL component, import the WebGL model integration package, write the Unity model running function, and implement the dexterous hand model integration window.

[0025] Preferably, in step S3, the backend server is built by the Django framework, the API interface and database architecture are designed, and the offline database is built. The specific process is as follows:

[0026] Step S31: Use the IDEA development tool to create a Django project and a project application; register the app and solve cross-domain issues, configure the database, and install project dependencies;

[0027] Step S32: design the interface path and bind the interface function; design the database table, establish the data structure, generate the database design operation and run the database table creation operation;

[0028] Step S33, implement interface business logic, including data access and hand motion recognition algorithm based on opencv-mediapipe;

[0029] Step S34: Use the Postman tool to perform interface testing.

[0030] Preferably, in step S4, based on the hand motion data captured in real time by the video window, image recognition and analysis technology is used to accurately map the hand motion to the 3D model, so as to realize the instant generation and transmission of motion control data. The specific process is as follows:

[0031] Step S41, calling the camera through the front end and sending a recognition request to the back end;

[0032] Step S42: The back-end hand motion recognition algorithm interface receives the data, performs motion recognition, generates motion control data, remotely controls the actual dexterous hand motion, and transmits the data to the Web through the serial port;

[0033] Step S43, the WebGL window receives serial port data through jslib and sends it to Unity, calls the model action function to control the model movement, returns the data to the front end, and calls the local backend interface to store it in the offline database.

[0034] Preferably, in step S5, human-computer interaction is designed to provide data security and standardization during serial port transmission to prevent data leakage and tampering. The specific process is as follows:

[0035] Step S51, designing the input format of motion control information, adding start and end characters, and preventing data leakage;

[0036] Step S52: On the Unity side, create a jslib file, establish a connection between Vue and WebGL, and implement functions for data interaction and event monitoring between Vue and jslib, jslib and Unity, and Unity and Vue;

[0037] Step S53, when jslib receives serial port data, increase the cache to read the data in lines. If the line is not full, continue to wait. If it overflows, use the queue to send.

[0038] Step S54: Add the function of format parsing of data when sending or receiving at other terminals.

[0039] Preferably, in step S6, the front-end interface uses the chart library function to visualize various types of data in the database, and updates the results in real time and displays them on the page. The specific process is as follows:

[0040] Step S61, bind the components in Vue with the requested data, and retain the data format by passing it through the parent-child components;

[0041] Step S62: Associating the data in the subcomponent with the Echarts chart settings, and dynamically updating the chart data using the watch() method;

[0042] Step S63: Add a button for displaying past data on the visualization interface, monitor click events, send offline data requests to the local backend, and return action trajectory data;

[0043] Step S64: the front end sends action control data to WebGL, and displays the model action of the past data in the 3D integrated window;

[0044] Step S65: The visualization interface synchronously displays the previous status data by monitoring events.

[0045] A dexterous hand digital twin system based on virtual-reality interaction, including a front-end interactive interface, back-end processing logic, a database management system and a digital twin model;

[0046] Among them, the front-end and back-end interaction framework consists of four core parts: front-end interaction layer, back-end logic layer, data management and storage layer, and visualization display layer.

[0047] Therefore, the present invention adopts the above-mentioned dexterous hand digital twin system based on virtual-reality interaction and its development method, and the beneficial effects are as follows:

[0048] (1) The present invention integrates the front-end page with UnityWebGL, and the project successfully builds a digital twin interface that can respond to hand movements in real time and drive the movement of the virtual model;

[0049] (2) The present invention utilizes the collaborative work of the Vue.js framework and the Django backend architecture to achieve rapid processing of operator instructions and real-time display of data, thereby improving the overall performance of the system;

[0050] (3) The combination of the MySQL database and the SQLyog management tool of the present invention not only ensures the storage and management of data, but also realizes the visualization of data on the front-end page through Echarts, thereby enhancing the readability and usability of data;

[0051] (4) The front-end and back-end separation architecture and rich plug-in support adopted by the present invention enable the system to run stably on different platforms and have good scalability, providing convenience for the addition of future functions;

[0052] (5) Through the responsive mechanism of Vue 3.0 and the asynchronous communication function of Axios, the system can intelligently respond to the operator's operation and update the status information of the dexterous hand in real time, ensuring the accuracy and timeliness of the information;

[0053] (6) The present invention integrates the video hand motion recognition function to achieve accurate capture, real-time analysis and remote control of dexterous hands, further improving the interaction accuracy and response speed of the digital twin system.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a method for developing a dexterous hand digital twin system based on virtual-reality interaction according to the present invention;

[0056] Figure 2 This is a flowchart for constructing a digital twin system of a dexterous hand based on virtual-reality interaction according to the present invention;

[0057] Figure 3 A flow chart of virtual model movement driven by data of the present invention;

[0058] Figure 4 Design a flow chart for human-computer interaction of the present invention;

[0059] Figure 5 This is the architecture diagram of the dexterous hand digital twin system of the present invention;

[0060] Figure 6 Schematic diagram of the digital twin model of the dexterous hand of the present invention;

[0061] Figure 7 It is a schematic diagram of the visualization interface of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0063] Example

[0064] like Figure 1 and Figure 2 As shown, the present invention provides a method for developing a dexterous hand digital twin system based on virtual-reality interaction, comprising the following steps:

[0065] Step S1: Use digital modeling tools to build a highly detailed dexterous hand model, which is bound with a detailed bone structure and rich action details to ensure the accuracy and authenticity of the model.

[0066] Step S11, obtaining the joints and module size and shape information of the actual dexterous hand, and establishing the corresponding component models on the Unity platform; combining the component models, distinguishing between free joints and fixed joints, and assembling to form a complete virtual dexterous hand;

[0067] Step S12: Build a scene in Unity according to the environment information and place the component model into the scene; add texture maps to all components and scenes, add material balls, assign physical properties, and form geometric modeling;

[0068] Step S13, creating a C# script file, defining components and life cycle functions; adding action parent-child relationships to the 3D model to ensure that the virtual dexterous hand moves in accordance with the laws of physical motion;

[0069] Step S14, design the variables that need to be controlled for the model movement, define parameters such as speed and acceleration, limit the range of movement, and ensure that the dexterous hand moves within a safe range;

[0070] Step S15, add a complete action control function to realize the control of model movement by input control information; add sliding components, bind active joints, and directly control the model movement through components.

[0071] Step S2: Design a visualization front-end page, which integrates a 3D model display area to instantly present the dynamic motion state of the dexterous hand. At the same time, the page also embeds a video acquisition window to capture and deeply analyze the operator's hand movements in real time.

[0072] Step S21, use IDEA development tool to create a Vue project and install project dependencies; introduce the Store module to implement global state management of the Vue project, and introduce the Element tool library as the front-end UI component library;

[0073] Step S22: introduce axios, configure asynchronous communication interceptor, build request framework; create request management js file, and globally manage all request configurations;

[0074] Step S23, create an application using the imported module and mount the Vue application;

[0075] Step S24, design the overall page and various components; introduce the Echarts drawing tool, and add a visual chart to display the dexterous hand model information;

[0076] Step S25, introduce the camera component, write the camera call method, and implement the camera integration window;

[0077] Step S26: Introduce the WebGL component, import the WebGL model integration package, write the Unity model running function, and implement the dexterous hand model integration window.

[0078] Step S3: The backend server is built with the Django framework, and the API interface and database architecture are designed to store the core physical information such as the structural parameters and motion trajectory of the dexterous hand, and to build an offline database.

[0079] Step S31: Use the IDEA development tool to create a Django project and a project application; register the app and solve cross-domain issues, configure the database, and install project dependencies;

[0080] Step S32: design the interface path and bind the interface function; design the database table, establish the data structure, generate the database design operation and run the database table creation operation;

[0081] Step S33, implement interface business logic, including data access and hand motion recognition algorithm based on opencv-mediapipe;

[0082] Step S34: Use tools such as Postman to perform interface testing.

[0083] Step S4: Based on the hand motion data captured in real time by the video window, image recognition and analysis technology is used to accurately map these hand movements to the 3D model, thereby realizing the instant generation and transmission of motion control data.

[0084] like Figure 3 As shown in the figure, first of all, when developing this digital twin system, Unity is selected to create a virtual dexterous hand model. Unity not only needs to have powerful 3D modeling capabilities, but also needs to support real-time data updates and physical simulation. In the model creation stage, each joint and movement of the dexterous hand is designed according to the above steps to ensure that it can achieve movements similar to those of a real hand in a virtual environment.

[0085] Secondly, the system accesses real-time data or past data to drive the movement of the virtual dexterous hand model. In this step, the operator can decide whether to use real-time data or past data. If real-time operation simulation or control is required, real-time data is selected; if analysis or training is performed, past data is more appropriate. Data can be obtained from data sources such as databases or cameras by calling interface requests.

[0086] After data acquisition, security permission verification is required to ensure the legitimacy and security of the data. Through this step, the system can effectively prevent unauthorized data access and operation, and protect operator privacy and data security.

[0087] Subsequently, the design of the integrated window of the virtual dexterous hand model is also a crucial step. The model is integrated into the web page, and data is sent to the model through the integrated window to control its movement. The implementation of this step depends on the close cooperation between the front-end and the back-end to ensure smooth data transmission and real-time update.

[0088] At the end of the entire process, the developed system will be fully tested to verify whether its functions are normal and whether its performance meets the standards. Through continuous optimization and adjustment, an efficient, stable and reliable digital twin system of dexterous hands based on virtual-real interaction will eventually be built.

[0089] The data-driven virtual model motion process is specifically implemented as follows:

[0090] Step S41, calling the camera through the front end and sending a recognition request to the back end;

[0091] Step S42: The back-end hand motion recognition algorithm interface receives the data, performs motion recognition, generates motion control data, remotely controls the actual dexterous hand motion, and transmits the data to the Web through the serial port;

[0092] Step S43, the WebGL window receives serial port data through jslib and sends it to Unity, calls the model action function to control the model movement, returns the data to the front end, and calls the local backend interface to store it in the offline database.

[0093] Step S5: design human-computer interaction to ensure data security and standardization during serial port transmission to prevent data leakage and tampering.

[0094] like Figure 4 As shown in the figure, the front-end and back-end work closely together to achieve efficient data transmission and processing, significantly improving the overall performance of the system. The front-end uses the Vue.js framework to design the operator interaction interface, integrates the Unity WebGL plug-in to display the virtual dexterous hand model, and uses Axios for asynchronous communication to ensure that the front-end module components can efficiently obtain data from the back-end and update in real time. The back-end uses the Django framework to build the business logic layer to handle complex data processing and storage tasks. The Django view layer provides API interfaces, and the front-end obtains hand motion control data and status data by calling these interfaces and displays them in real time on the page.

[0095] At the same time, the backend is also responsible for storing this data in the MySQL database to support offline data calls and displays. Through the dynamic requests of the front end and the instant response of the back end, the data is seamlessly connected between the virtual and real space, which not only ensures the accuracy and real-time nature of the data, but also greatly improves the response speed and operating efficiency of the system, providing operators with a smooth operating experience.

[0096] The human-computer interaction design process, the specific implementation process is as follows:

[0097] Step S51, designing the input format of motion control information, adding start and end characters, and preventing data leakage;

[0098] Step S52: On the Unity side, create a jslib file, establish a connection between Vue and WebGL, and implement functions for data interaction and event monitoring between Vue and jslib, jslib and Unity, and Unity and Vue;

[0099] Step S53, when jslib receives serial port data, increase the cache to read the data in lines. If the line is not full, continue to wait. If it overflows, use the queue to send.

[0100] Step S54: Add the function of format parsing of data when sending or receiving at other terminals.

[0101] Step S6: The front-end interface uses the chart library function to visualize various types of data in the database, and updates the results in real time and displays them on the page.

[0102] Step S61, bind the components in Vue with the requested data, and retain the data format by passing it through the parent-child components;

[0103] Step S62: Associating the data in the subcomponent with the Echarts chart settings, and dynamically updating the chart data using the watch() method;

[0104] Step S63: Add a button for displaying past data on the visualization interface, monitor click events, send offline data requests to the local backend, and return action trajectory data;

[0105] Step S64: the front end sends action control data to WebGL, and displays the model action of the past data in the 3D integrated window;

[0106] Step S65: The visualization interface synchronously displays the previous status data by monitoring events.

[0107] Among them, the digital twin model of the dexterous hand and the schematic diagram of the visualization interface are as follows: Figure 6 and Figure 7 shown.

[0108] Based on the above system development method, the present invention provides a dexterous hand digital twin system based on virtual-real interaction. The digital twin system architecture is as follows: Figure 5 shown.

[0109] Digital twin technology enables real-time monitoring of the system by creating a virtual model corresponding to the actual physical system. The system architecture shows the relationship and interaction between the various components. As the core part, the digital twin model accurately models the movement and state of the actual dexterous hand, allowing the operator to simulate the operation in a virtual environment. The human-computer interactive digital twin system enables the operator to interact with the virtual model, send instructions through the front-end interface and obtain system responses. As the basic unit, motion control data is stored and managed through a database management system, recording every movement and state change of the dexterous hand, providing a basis for precise control. The back-end processing logic processes this data and predicts and controls the movement of the dexterous hand through complex algorithms and models. The front-end interactive interface provides an intuitive operating experience, allowing the operator to monitor the state of the dexterous hand in real time. The close interaction between the digital twin model and the interactive system ensures the accuracy and real-time performance of the virtual model, which directly affects the operator experience and analysis results.

[0110] According to the various physical characteristics of the dexterous hand, the system constructs the front-end interactive interface, back-end processing logic, database management system and digital twin model, and generates the twin model of the dexterous hand by associating and binding the above models. At the same time, the corresponding motion control rules are formulated, the dexterous hand system information library is established, and the virtual and real data interaction strategy is used to realize the dexterous hand data visualization and data transmission, and complete the simulation, driving and monitoring functions of the dexterous hand. It mainly realizes the following functions:

[0111] Build a digital twin model of the dexterous hand to realize the action functions of the dexterous hand and the digital twin model of the dexterous hand; interact with the system intuitively through the front-end page, view the system status in real time, and realize the call and display of offline data; build a digital twin system for virtual and real interaction to identify the operator's hand movements in real time, and drive the virtual model to perform corresponding movements accordingly; build a dexterous hand database and design a database management system that includes the digital twin model and the physical information of the dexterous hand; the front-end and back-end work closely together to ensure efficient transmission and processing of data and improve the overall performance of the system.

[0112] Among them, the Unity digital modeling tool is used to create a virtual model of the dexterous hand to ensure that its structure is highly consistent with the physical dexterous hand. On the Unity side, the motion control rules drive the virtual model to perform corresponding movements by receiving motion control data from the front end. In order to achieve high-fidelity mirroring, the present invention also designs a set of efficient communication mechanisms to ensure that data in the virtual and real spaces can be transmitted interactively in real time and accurately. The C# script code in Unity defines the motion logic of the model in detail, including parameters such as the rotation angle and speed of each joint. These parameters are dynamically adjusted according to the received motion control data to achieve accurate simulation of hand movements by the virtual model.

[0113] In terms of specific implementation, the rules define multiple joint objects and their corresponding center points and rotation speeds. After receiving the hand motion data, the script calculates the target angle of each joint in the virtual dexterous hand model and dynamically adjusts the rotation speed according to the difference between the current angle and the target angle. In this process, a proportional control strategy is adopted to ensure that the movement of the virtual model is both smooth and responsive to hand movements.

[0114] In addition, the rules also include the definition of speed range (such as Hs and Ls), which is used to limit the maximum and minimum speeds of joint rotation to prevent the model from excessive movement or damage due to receiving abnormal data. By judging the angle difference, when the virtual model approaches the target angle, the rotation speed will be gradually reduced until it stops, achieving precise positioning. At the same time, in order to ensure the security of data transmission, the present invention adopts a verification mechanism during the data transmission process to effectively prevent the risk of data tampering or loss.

[0115] The front-end page serves as an intuitive window for the operator to interact with the system, integrating rich functional designs. The operator can not only view the motion status of the dexterous hand in real time through the camera window on the page, but also use the operation buttons and control panel on the page to flexibly adjust the system parameters and configurations. At the same time, the front-end page displays the motion data of the dexterous hand in real time, including key information such as the angle and speed of each joint. These data are presented in an intuitive graphical way through the chart library, allowing the operator to grasp the current status of the system at a glance. In addition, the front-end page also supports seamless docking with the database, and the operator can easily call historical data for comparative analysis, so as to have a more comprehensive understanding of the operation status of the dexterous hand.

[0116] The construction of a digital twin system for virtual-reality interaction realizes real-time recognition of the operator's hand movements and synchronous driving of the virtual model. Using Python's opencv-mediapipe algorithm, the system can accurately capture subtle changes in hand movements and convert them into detailed motion control data. These data are transmitted to the Unity platform in real time through an efficient communication mechanism, driving the virtual model to move synchronously with the physical dexterous hand.

[0117] The database uses MySQL as a storage solution, using its high performance and reliability to ensure the stability and security of system data. In the design of database tables, special data tables are designed for the hand motion data, dexterous hand status information and various data generated by system interactions that the system needs to store. These table structures are planned and contain necessary fields to comprehensively record relevant data. For example, the table contains the unique identifier id, name name and specific value of physical information items, ensuring the integrity and accuracy of the data. The database design fully considers the relevance and query efficiency of the data. By defining database structures such as primary keys and indexes, it not only ensures the logical connection between data, but also improves the speed of data retrieval. At the same time, in order to facilitate data management and maintenance, the Django ORM framework is used for database operations, which simplifies the process of adding, deleting, modifying and querying data and improves development efficiency.

[0118] The front-end and back-end interaction framework consists of four core parts: the front-end interaction layer, the back-end logic layer, the data management and storage layer, and the visualization layer. The framework realizes intuitive interaction between the operator and the system through the front-end interaction layer, while the back-end logic layer is responsible for processing complex business logic and data transmission to ensure the real-time and accuracy of the data. The data management and storage layer efficiently manages and persistently stores the massive data generated by the system, providing data support. The visualization layer presents the virtual dexterous hand model in an intuitive and realistic way, realizing real-time synchronization and interaction between the dexterous hand entity and the virtual model, thereby building a highly integrated and fully functional dexterous hand digital twin system.

[0119] Therefore, the present invention adopts the above-mentioned digital twin system of dexterous hands based on virtual-reality interaction and its development method, and realizes high-precision and real-time capture of hand movements by integrating computer vision and deep learning technology, effectively solving the problems of low precision and susceptibility to environmental interference of traditional technologies, and improving the accuracy and realism of virtual model movements; at the same time, using digital twin technology, a physical mapping relationship between the hand model and the virtual model is constructed to achieve real-time synchronization and interaction between hand movements and the virtual environment, bringing a more intuitive and realistic operation experience to the operator; in addition, the system also introduces visualization technology, which comprehensively displays the subtle differences and dynamic changes of hand movements through various intuitive forms such as charts and animations, greatly helping the operator to better understand and control the movement of the virtual model.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for developing a digital twin system of a dexterous hand based on virtual-reality interaction, characterized in that: The following steps are involved: Step S1, using digital modeling tools to create a dexterous hand model, and binding the bone structure and motion information to it; Step S2, designing a visualization front-end page, which integrates a 3D model display window for real-time presentation of the motion state of the dexterous hand; and embeds a video acquisition window to capture and analyze hand movements; Step S3: The backend server is built with the Django framework, and the backend interface and database architecture are designed to save the key physical information of the dexterous hand and build an offline database; Step S4: Based on the hand movements captured by the video window, image recognition and analysis technology is used to accurately map the movements of the 3D model, thereby achieving real-time generation of motion control data; Step S5: design human-computer interaction to ensure data security and standardization during serial port transmission to prevent data leakage and tampering; Step S6: The front end uses the chart library to visualize the data in the database and displays it on the page, while updating the model motion status and joint angle data in real time.

2. The method for developing a digital twin system of a dexterous hand based on virtual-reality interaction according to claim 1 is characterized in that: In step S1, a dexterous hand model is constructed using a digital modeling tool. The specific process is as follows: Step S11, obtaining the joints and module size and shape information of the actual dexterous hand, and establishing the corresponding component models on the Unity platform; combining the component models, distinguishing between free joints and fixed joints, and assembling to form a complete virtual dexterous hand; Step S12: Build a scene in Unity according to the environment information and place the component model into the scene; add texture maps to all components and scenes, add material balls, assign physical properties, and form geometric modeling; Step S13, creating a C# script file, defining components and life cycle functions; adding action parent-child relationships to the 3D model to ensure that the virtual dexterous hand moves in accordance with the laws of physical motion; Step S14, design the variables that need to be controlled for the model movement, define the speed and acceleration parameters; limit the movement range to ensure that the dexterous hand movements are within a safe range; Step S15, add a complete action control function to realize the control of model movement by input control information; add sliding components, bind active joints, and directly control the model movement through components.

3. The method for developing a dexterous hand digital twin system based on virtual-reality interaction according to claim 1 is characterized in that: In step S2, a visualization front-end page is designed, which integrates a 3D model display area and embeds a video acquisition window. The specific process is as follows: Step S21, use IDEA development tool to create a Vue project and install project dependencies; introduce the Store module to implement global state management of the Vue project, and introduce the Element tool library as the front-end UI component library; Step S22: introduce axios, configure asynchronous communication interceptor, and build request framework; Create a request management js file to globally manage all request configurations; Step S23, create an application using the imported module and mount the Vue application; Step S24, designing the overall page and each component; Introduced Echarts drawing tool and added visual charts for displaying dexterous hand model information; Step S25, introduce the camera component, write the camera call method, and implement the camera integration window; Step S26: introduce the WebGL component, import the WebGL model integration package, write the Unity model running function, and implement the dexterous hand model integration window.

4. The method for developing a digital twin system of a dexterous hand based on virtual-reality interaction according to claim 1 is characterized in that: In step S3, the backend server is built with the Django framework, the API interface and database architecture are designed, and the offline database is built. The specific process is as follows: Step S31: Use the IDEA development tool to create a Django project and a project application; register the app and solve cross-domain issues, configure the database, and install project dependencies; Step S32: design the interface path and bind the interface function; design the database table, establish the data structure, generate the database design operation and run the database table creation operation; Step S33, implementing interface business logic, including data access and hand motion recognition algorithm based on opencv-mediapipe; Step S34: Use the Postman tool to perform interface testing.

5. The method for developing a dexterous hand digital twin system based on virtual-reality interaction according to claim 1 is characterized in that: In step S4, based on the hand motion data captured in real time by the video window, image recognition and analysis technology is used to accurately map the hand motion to the 3D model, thereby realizing the instant generation and transmission of motion control data. The specific process is as follows: Step S41, calling the camera through the front end and sending a recognition request to the back end; Step S42: The back-end hand motion recognition algorithm interface receives the data, performs motion recognition, generates motion control data, remotely controls the actual dexterous hand motion, and transmits the data to the Web through the serial port; Step S43, the WebGL window receives serial port data through jslib and sends it to Unity, calls the model action function to control the model movement, returns the data to the front end, and calls the local backend interface to store it in the offline database.

6. The method for developing a dexterous hand digital twin system based on virtual-reality interaction according to claim 1 is characterized in that: In step S5, human-computer interaction is designed to ensure data security and standardization during serial port transmission to prevent data leakage and tampering. The specific process is as follows: Step S51, designing the input format of motion control information, adding start and end characters, and preventing data leakage; Step S52: On the Unity side, create a jslib file, establish a connection between Vue and WebGL, and implement functions for data interaction and event monitoring between Vue and jslib, jslib and Unity, and Unity and Vue; Step S53, when jslib receives serial port data, increase the cache to read the data in lines. If the line is not full, continue to wait. If it overflows, use the queue to send. Step S54: Add the function of format parsing of data when sending or receiving at other terminals.

7. The method for developing a digital twin system of a dexterous hand based on virtual-reality interaction according to claim 1 is characterized in that: In step S6, the front-end interface uses the chart library function to visualize various types of data in the database, and updates the results in real time and displays them on the page. The specific process is as follows: Step S61, bind the components in Vue with the requested data, and retain the data format by passing it through the parent-child components; Step S62: Associating the data in the subcomponent with the Echarts chart settings, and dynamically updating the chart data using the watch() method; Step S63: Add a button for displaying past data on the visualization interface, monitor click events, send offline data requests to the local backend, and return action trajectory data; Step S64: the front end sends action control data to WebGL, and displays the model action of the past data in the 3D integrated window; Step S65: The visualization interface synchronously displays the previous status data by monitoring events.

8. According to the method for developing a dexterous hand digital twin system based on virtual-reality interaction according to any one of claims 1 to 7, a dexterous hand digital twin system based on virtual-reality interaction is developed, characterized in that: The system includes: front-end interactive interface, back-end processing logic, database management system and digital twin model; Among them, the front-end and back-end interaction framework consists of four core parts: front-end interaction layer, back-end logic layer, data management and storage layer, and visualization display layer.

Images