A virtual haptic interaction method and system for cultural relics based on multi-modal perception fusion

Abstract

This application provides a method and system for virtual tactile interaction of cultural relics based on multimodal perception fusion. The method includes: preprocessing a 3D ancient scene and a 3D digital model of the cultural relic corresponding to a physical cultural relic model; transmitting one of the physical cultural relic models of different materials to a designated interactive area according to the progress of content preview; identifying the spatial information of the physical cultural relic model using spatial positioning technology, and fusing the spatial information of the physical cultural relic model with the pose of a virtual camera in a VR headset; activating a gesture tracking device to perform real-time gesture tracking, acquiring real-time gesture tracking data, and rendering a virtual hand in the virtual space based on the real-time gesture tracking data; when a viewer wears VR headsets to observe the virtual scene, while the real hand touches the physical cultural relic model, the virtual hand simultaneously touches the virtual cultural relic model in the virtual space. This application improves the accuracy of virtual-real pose fusion during virtual cultural relic interaction, reduces interaction response latency, and provides a better user experience.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a method and system for virtual tactile interaction of cultural relics based on multimodal perception fusion. Background Technology

[0002] With increasing awareness of cultural heritage protection and the development of digital twin technology, virtual museums and digital artifact display systems have been widely adopted. Existing technologies primarily focus on the visual reconstruction and display of 3D models of artifacts, such as constructing high-precision geometric models based on structured light scanning or photogrammetry, and combining this with PBR (Physically Based Rendering) technology to achieve realistic visual effects. However, touch, as a crucial channel for humans to perceive the material world, is severely lacking in existing virtual interactive systems. Users cannot perceive the surface roughness, hardness, texture, temperature, and other physical properties of artifacts, resulting in an interactive experience that remains at a superficial "see but not touch" stage, failing to meet the in-depth needs of scenarios such as archaeological research, artifact appreciation, and science education.

[0003] Existing virtual museums and digital artifact display systems focus solely on 3D visual reconstruction. Users cannot perceive the physical properties of artifacts, such as surface roughness, hardness, texture, and temperature, resulting in a superficial interactive experience. Traditional systems require actual user contact before initiating tactile feedback, leading to significant delays (slow response times) and a disconnect between visual and tactile feedback. Furthermore, the lack of anticipation of user interaction intentions prevents proactive and immediate feedback.

[0004] Therefore, the urgent technical problem to be solved is: how to provide a method and system for virtual tactile interaction of cultural relics based on multimodal perception fusion, so as to solve the problems of low accuracy, high latency and shallow experience of virtual and real pose fusion in the process of virtual cultural relic interaction. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for virtual tactile interaction of cultural relics based on multimodal perception fusion, which improves the accuracy of virtual and real pose fusion during virtual cultural relic interaction, reduces interaction response latency, and provides a better user experience.

[0006] To achieve the above objectives, as a first aspect of this application, this application provides a method for virtual tactile interaction of cultural relics based on multimodal perception fusion. The method includes: preprocessing a three-dimensional ancient scene and a three-dimensional digital model of the cultural relic corresponding to a physical cultural relic model; transmitting one of the physical cultural relic models of different materials to a designated interactive area according to the progress of content preview; identifying the spatial information of the physical cultural relic model using spatial positioning technology, and fusing the spatial information of the physical cultural relic model with the pose of a virtual camera in a VR glasses device; activating a gesture tracking device to perform real-time gesture tracking, acquiring real-time gesture tracking data, and rendering a virtual hand in virtual space based on the real-time gesture tracking data; and providing a tactile interactive experience between the virtual digital content and the physical cultural relic model.

[0007] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, includes the preprocessing of a 3D ancient scene and a 3D digital model of the cultural relic corresponding to the physical cultural relic model, which includes: Collect attribute and characteristic information of physical cultural relic models; Based on the historical background and excavation environment of the physical artifact model, the corresponding three-dimensional ancient scene is called up through modeling software; Based on the material of the physical artifact model, a three-dimensional digital model of the artifact is generated using modeling software, matching the scale of the physical artifact model.

[0008] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, includes the attribute feature information of the physical cultural relic model, including historical background, excavation environment, and material.

[0009] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, includes transmitting one of the physical cultural relic models of different materials to the designated interactive area according to the progress of content previewing. Initiate the content preview program for cultural relic interaction. Through the progress node markers within the content preview program, identify the current progress stage of the content preview in real time and determine the type of cultural relic material to be displayed at the current progress stage of the content preview. Based on the determined material type of the cultural relic, retrieve the corresponding three-dimensional digital model of the cultural relic from the physical cultural relic model storage area; The physical artifact model is transported to the designated interactive area directly in front of the audience using a mechanical conveyor system.

[0010] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, includes activating a gesture tracking device to perform real-time gesture tracking, acquiring real-time gesture tracking data, and rendering a virtual hand in the virtual space based on the real-time gesture tracking data, which includes:

[0011] Activate the gesture tracking device and determine the spatial range for gesture acquisition; Based on the spatial range of gesture acquisition, real-time gesture tracking is performed, and real-time gesture tracking data is collected. Real-time gesture tracking data is mapped onto a 3D virtual hand model to generate a virtual hand in virtual space.

[0012] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, includes real-time gesture tracking based on the spatial range of gesture acquisition. After collecting real-time gesture tracking data, it also includes: confidence scoring of multiple gesture acquisition devices, and optimization of the data collected by the gesture acquisition devices based on the confidence scores.

[0013] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, includes the following steps: calculating the confidence scores of multiple gesture acquisition devices and optimizing the data collected by these devices based on these confidence scores. Acquire confidence evaluation index data for gesture acquisition devices; Based on the measurement variance of the gesture acquisition device, calculate the confidence score of the gesture acquisition device, and generate a standardized data stream carrying the confidence score of the gesture acquisition device; The data in the standardized data stream is optimized based on the confidence score of the gesture acquisition device.

[0014] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, involves generating a virtual hand and then optimizing the rendering of the virtual hand.

[0015] The virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described above, wherein, Spatial registration technology is used to establish a spatial mapping relationship between virtual and physical artifact models, enabling real-time synchronization between the virtual artifact's pose in virtual space and the physical artifact's pose in real space. When a viewer wears VR glasses to observe a virtual scene, while their real hand touches the physical artifact model in a designated interactive area, their virtual hand simultaneously touches the corresponding virtual artifact model in virtual space, including: Establish a spatial mapping relationship between virtual cultural relic models and physical cultural relic models to ensure that the position, posture, and size of virtual cultural relics in VR space are synchronized with the actual posture of physical cultural relics in physical space in real time. Real-time detection of the physical contact between the viewer's real hand and the physical artifact model, and synchronous triggering of virtual hand-to-art touch events between the virtual hand and the virtual artifact model; Based on the location and type of physical touch, multimodal interactive feedback is rendered in the VR virtual space to enhance the unified experience of vision and touch.

[0016] As a second aspect of this application, this application provides a virtual tactile interaction system for cultural relics based on multimodal perception fusion, the system comprising: The preprocessing module is used to preprocess the 3D ancient scene and 3D digital model of the cultural relic corresponding to the physical cultural relic model; The delivery module is used to deliver one of the physical artifact models of different materials to the designated interactive area according to the progress of the content preview. The fusion module is used to identify the spatial information of the physical cultural relic model using spatial positioning technology, and then fuse the spatial information of the physical cultural relic model with the pose of the virtual camera in the VR glasses device. Gesture tracking devices perform real-time gesture tracking and acquire real-time gesture tracking data; The rendering module is used to render a virtual hand in virtual space based on real-time gesture tracking data; The interaction module is used to establish a spatial mapping relationship between virtual cultural relic models and physical cultural relic models through spatial registration technology, so that the pose of virtual cultural relics in virtual space is synchronized with the pose of physical cultural relics in real space in real time. When the audience wears VR glasses to observe the virtual scene, while their real hand touches the physical cultural relic model in the designated interactive area, their virtual hand simultaneously touches the corresponding virtual cultural relic model in the virtual space.

[0017] The beneficial effects achieved by this application are as follows: (1) This application transmits the physical cultural relic model to the designated interactive area in front of the audience, ensuring that the physical anchor point of the physical cultural relic model coincides with the spatial positioning reference point of the designated interactive area, and that the posture of the physical cultural relic model remains stable.

[0018] (2) This application eliminates the fusion error between the three-dimensional spatial pose data of the physical cultural relic model and the real-time pose data of the virtual camera by using an error compensation algorithm, thereby improving the accuracy of virtual-real pose fusion.

[0019] (3) This application scores the confidence of multiple gesture acquisition devices, and optimizes the data collected by the gesture acquisition devices based on the confidence scores to improve the accuracy of the data collected by the gesture acquisition devices.

[0020] (4) This application maps the collected hand key point tracking data to a pre-made three-dimensional virtual hand model to achieve real-time synchronization of the virtual hand position, posture, finger flexion and extension movements with the audience's real gestures without delay. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart illustrating a virtual tactile interaction method for cultural relics based on multimodal perception fusion, as described in an embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the structure of a virtual tactile interaction system for cultural relics based on multimodal perception fusion, according to an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0025] like Figure 1 As shown, this application provides a virtual tactile interaction method for cultural relics based on multimodal perception fusion, the method comprising: Step S1: Preprocess the 3D ancient scene and 3D digital model of the cultural relic corresponding to the physical cultural relic model.

[0026] Step S1 includes: Step S110: Collect attribute feature information of the physical cultural relic model.

[0027] The attribute characteristics of the physical artifact model include information such as historical background, excavation environment, and materials.

[0028] Step S120: Based on the historical background and excavation environment of the physical cultural relic model, the modeling software calls up the corresponding three-dimensional ancient scene.

[0029] Modeling software such as 3ds Max and Blender.

[0030] Step S130: Based on the material of the physical cultural relic model, a three-dimensional digital model of the cultural relic is generated using modeling software that matches the physical cultural relic model in proportion.

[0031] Specifically, 3D digital models of the cultural relics are created using modeling software, corresponding to the physical artifacts at a 1:1 scale. The materials used for the physical artifact models include bronze, jade, and ceramics.

[0032] Step S2: Based on the progress of the content preview, send one of the physical artifact models of different materials to the designated interactive area.

[0033] Specifically, depending on the progress of the content preview, one of the physical artifact models made of different materials will be transported to the front of the audience.

[0034] Step S2 includes: Step S210: Start the content preview program for cultural relic interaction. Through the progress node markers in the content preview program, identify the current progress stage of the content preview in real time and determine the type of cultural relic material to be displayed at the current progress stage of the content preview.

[0035] Step S220: Based on the determined material type of the cultural relic, retrieve the corresponding three-dimensional digital model of the cultural relic from the physical cultural relic model storage area.

[0036] Step S230: The physical artifact model is transported to the designated interactive area directly in front of the audience via a mechanical conveyor.

[0037] The process involves transmitting a physical artifact model to a designated interactive area directly in front of the audience, ensuring that the physical anchor point of the artifact model coincides with the spatial positioning reference point of the interactive area, and that the artifact model maintains a stable posture.

[0038] Step S3: Use spatial positioning technology to identify the spatial information of the physical cultural relic model, and then fuse the spatial information of the physical cultural relic model with the pose of the virtual camera in the VR glasses device.

[0039] Step S3 includes: Step S310: Identify the three-dimensional spatial pose data of the physical cultural relic model using a spatial positioning device.

[0040] The three-dimensional spatial information of the physical artifact model includes position coordinates (X / Y / Z axes) and attitude angles (yaw angle / pitch angle / roll angle), and the collected pose data is transmitted to the system processing terminal in real time at a preset frequency.

[0041] Spatial positioning equipment includes, for example, visual SLAM equipment, infrared positioning equipment, and laser positioning equipment.

[0042] Step S320: Obtain the real-time pose information of the virtual camera in the VR glasses device worn by the viewer.

[0043] The real-time pose information of the virtual camera in the VR glasses device includes: position and posture. The real-time pose information of the virtual camera in the VR glasses device reflects the viewer's observation angle and position, and is transmitted to the system processing terminal in real time.

[0044] Step S330: Perform pre-fusion calibration on the three-dimensional spatial pose data of the collected physical artifact model and the real-time pose data of the virtual camera. Eliminate the spatial deviation between the two types of data through coordinate transformation and error compensation algorithms to achieve registration and alignment of virtual and real pose data.

[0045] Specifically, the system processing unit performs pre-fusion calibration on the three-dimensional spatial pose data of the collected physical artifact model and the real-time pose data of the virtual camera. Through coordinate transformation and error compensation algorithms, it eliminates the spatial deviation between the two types of data and achieves the registration and alignment of virtual and real pose data.

[0046] In particular, the pose is uniformly represented by a homogeneous transformation matrix in the virtual-real fusion scene.

[0047] Homogeneous pose transformation matrix ; R: 3×3 rotation matrix, representing the attitude (pitch, yaw, roll) of the object / camera.

[0048] A 3×1 translation vector, representing the spatial coordinates.

[0049] The physical pose of the physical artifact model is acquired from a physical positioning coordinate system (such as the world coordinate system of a spatial positioning device), and needs to be converted to the VR virtual coordinate system. The conversion formula is as follows: ; in, This is the target pose matrix of the physical model in the virtual coordinate system (aligned with the VR scene). This is the transformation matrix from the virtual coordinate system to the physical coordinate system (obtained from system pre-calibration / real-time calibration, core calibration parameter). The original pose matrix of the physical artifact model in the physical coordinate system (acquired by spatial positioning equipment).

[0050] The objective function is to minimize the position error of the physical / virtual points. The objective function is as follows: ; in, The objective function is... For the model of physical cultural relics The physical coordinates of each feature point; The virtual cultural relic model is represented by the first The virtual coordinates of each anchor point; This indicates the number of feature points for virtual-real matching (≥3, ensuring uniqueness of the solution); min indicates taking the minimum value.

[0051] The optimal rotation matrix R and translation vector t are obtained from the objective function, and then substituted into the pose homogeneous transformation matrix to obtain T.

[0052] Among them, eliminating the fusion error between the three-dimensional spatial pose data of the physical cultural relic model and the real-time pose data of the virtual camera through the error compensation algorithm includes: correcting the fused pose through the error compensation function.

[0053] The error compensation function is as follows: ; in, This represents the error compensation value calculated using the error compensation function; This indicates the confidence level of the physical model's pose acquisition. This indicates the confidence level of the VR virtual camera pose (output from the head-mounted display).

[0054] The corrected pose is: ;in, This indicates the corrected pose. This represents the error compensation value calculated using the error compensation function. This represents the uncompensated original fused pose.

[0055] Step S340: Map the 3D spatial pose data of the registered physical artifact model to the VR virtual scene so that the pose of the 3D digital artifact model is completely matched with the pose of the physical artifact model.

[0056] Specifically, the 3D spatial pose data of the registered physical artifact model is mapped into the VR virtual scene, so that the pose of the 3D digital artifact model is completely matched with the pose of the physical artifact model. At the same time, based on the real-time pose data of the virtual camera, the 3D ancient scene corresponding to the physical text model is rendered and fused in real time in the VR glasses, so that the viewer's perspective of observing the physical artifact model is consistent with the perspective of the VR virtual scene.

[0057] Step S4: Start the gesture tracking device to perform real-time gesture tracking, acquire real-time gesture tracking data, and render a virtual hand in the virtual space based on the real-time gesture tracking data.

[0058] The gesture tracking device of this application performs real-time gesture tracking, renders a virtual palm in virtual space, realizes real-time mapping from real gestures to virtual palm, and provides an operation entry point for subsequent touch interaction.

[0059] Step S4 includes: Step S410: Start the gesture tracking device and determine the spatial range for gesture acquisition.

[0060] Specifically, activate the gesture tracking device (such as a motion-sensing depth camera, data gloves, or infrared gesture sensor), complete the connection and calibration between the device and the VR system, determine the spatial range of gesture acquisition (covering the interactive area directly in front of the viewer), and calibrate the recognition accuracy of key hand points.

[0061] Step S420: Based on the spatial range of gesture acquisition, perform real-time gesture tracking and collect real-time gesture tracking data.

[0062] Specifically, the gesture acquisition device detects key feature points of the audience's hands in real time (such as the center of the palm, finger joints, and fingertips), obtains the three-dimensional spatial coordinates and motion trajectory of each key point, and transmits the real-time gesture tracking data to the system rendering end at a high frame rate (≥60fps).

[0063] The gesture acquisition device includes multiple components, employing: 2 depth cameras deployed in front of and to the side of the audience (covering the front and side of the hands); 3-4 depth cameras deployed in a surround manner (covering 360° hand posture with no blind spots); and depth cameras and data gloves: the main device performs global tracking, while the gloves provide fine joint tracking.

[0064] As a specific embodiment of the present invention, the confidence scores of multiple gesture acquisition devices (representing the reliability of the detection results) are calculated, and the data collected by the gesture acquisition devices are optimized based on the confidence scores to ensure the stability, accuracy and smoothness of the interaction in the cultural relic VR virtual-real fusion interactive system.

[0065] As a specific embodiment of the present invention, real-time gesture tracking is performed based on the spatial range of gesture acquisition. After collecting real-time gesture tracking data, the method further includes: scoring the confidence of multiple gesture acquisition devices, and optimizing the data collected by the gesture acquisition devices based on the confidence scores. Scoring the confidence of multiple gesture acquisition devices and optimizing the data collected by the gesture acquisition devices based on the confidence scores includes: Step S421: Obtain the confidence evaluation index data of the gesture acquisition device.

[0066] The confidence evaluation index data for gesture acquisition devices include the measurement variance and operating temperature of the gesture acquisition devices.

[0067] Step S422: Calculate the confidence score of the gesture acquisition device based on the measurement variance of the gesture acquisition device, and generate a standardized data stream carrying the confidence score of the gesture acquisition device.

[0068] The standardized data stream contains a confidence score for the gesture acquisition device.

[0069] The formula for calculating the confidence score of the gesture acquisition device is as follows: ; in, This represents the confidence score of the i-th gesture acquisition device; This represents the measurement variance of the i-th gesture acquisition device; Let represent the measurement variance of the j-th gesture acquisition device. This indicates the total number of gesture capture devices. This represents the operating temperature of the i-th gesture acquisition device.

[0070] Step S423: Optimize the data in the standardized data stream based on the confidence score of the gesture acquisition device.

[0071] Based on the confidence scores of the gesture acquisition devices, the data in the standardized data stream is optimized by: pre-setting a mapping relationship between the optimization coefficients and the confidence score ranges of the corresponding gesture acquisition devices. Different confidence score ranges for different gesture acquisition devices correspond to different optimization coefficients (the optimization coefficients range from 0 to 1). The higher the confidence score of the gesture acquisition device, the larger the corresponding optimization coefficient. A matching optimization coefficient is selected based on the confidence score of the gesture acquisition device. The selected optimization coefficient is multiplied by the data in the standardized data stream to obtain the optimized data. The optimized data is then aggregated to form real-time gesture tracking data.

[0072] Step S430: Map the real-time gesture tracking data onto the three-dimensional virtual hand model to generate a virtual hand in the virtual space.

[0073] Specifically, the system rendering end maps the collected hand key point tracking data to a pre-made 3D virtual hand model, realizing real-time synchronization of the virtual hand's position, posture, finger flexion and extension movements with the audience's real gestures, without delay.

[0074] Step S440: Render the virtual hand.

[0075] Specifically, the virtual hand model rendered and mapped in the VR virtual space optimizes the visual effect of the virtual hand according to the lighting and texture style of the virtual scene, while ensuring that the rendering level of the virtual hand is compatible with the virtual scene and the digital model of the 3D cultural relic, without occlusion or clipping.

[0076] Step S5: Establish a spatial mapping relationship between the virtual cultural relic model and the physical cultural relic model through spatial registration technology, so that the pose of the virtual cultural relic in the virtual space is synchronized with the pose of the physical cultural relic in the real space in real time; when the audience wears VR glasses to observe the virtual scene, while their real hand touches the physical cultural relic model in the designated interactive area, the virtual hand simultaneously touches the corresponding virtual cultural relic model in the virtual space.

[0077] Step S5 includes: Step S510: Establish a spatial mapping relationship between the virtual cultural relic digital model and the physical cultural relic model to ensure that the position, posture, and size of the virtual cultural relic in VR space are synchronized with the actual posture of the physical cultural relic in physical space in real time.

[0078] Specifically, based on the fused pose data in step S3, a precise spatial correspondence is established between the virtual interactive area of ​​the virtual cultural relic digital model and the physical surface of the physical cultural relic model, so that the virtual surface of the virtual cultural relic coincides with the physical surface of the physical cultural relic in space, and the error is controlled within the tactile perception threshold.

[0079] Step S520: Real-time detection of the physical contact between the viewer's real hand and the physical artifact model, and synchronous triggering of virtual touch events between the virtual hand and the virtual artifact digital model.

[0080] Specifically, by deploying an array of tactile sensors in the designated interactive area or predicting hand positions based on gesture tracking data, the system can identify in real time the physical touch actions (including touch position, touch force, and gesture shape) of the viewer's real hand on the physical artifact model. When a physical touch is detected, the system simultaneously triggers a virtual touch effect between the virtual palm and the virtual artifact digital model in the corresponding spatial position in the VR virtual space.

[0081] Step S530: Render multimodal interactive feedback in the VR virtual space based on the location and type of physical touch to enhance the unified experience of vision and touch.

[0082] Specifically, based on the identified physical touch location and gesture type (such as tapping, stroking, holding, and knocking), corresponding visual feedback (such as highlighting the touch point, magnifying the details of the artifact, and unfolding historical scenes) is rendered in the virtual space, as well as auditory feedback (such as tactile sound effects corresponding to the material of the artifact) and virtual mechanical feedback (such as virtual hand deformation). This allows viewers to experience the physical tactile sensation (hardness, temperature, texture, and weight) of the artifact with their real hands while their eyes see a completely synchronized virtual interactive effect, eliminating the disconnect between visual and tactile perception.

[0083] Step S540 involves real-time monitoring and calibration of the entire process of virtual-real synchronous interaction to ensure a high degree of consistency between visual presentation and physical touch.

[0084] Specifically, the system monitors in real time the synchronization accuracy between the virtual hand pose and the real hand pose, as well as the spatial registration accuracy between the virtual artifact model and the physical artifact model. It corrects the virtual and real pose deviations through a dynamic error compensation algorithm and handles abnormal situations (such as hand tracking loss or artifact model displacement) to ensure a continuous and stable unified visual and tactile experience.

[0085] Users are unaware of the existence of real cultural relics. Through precise spatial registration, virtual vision and real touch are perfectly integrated, achieving a unified perceptual experience across modalities.

[0086] like Figure 2 As shown, this application provides a virtual tactile interaction system 100 for cultural relics based on multimodal perception fusion. The system includes: The preprocessing module 10 is used to preprocess the three-dimensional ancient scene and the three-dimensional digital model of the cultural relic corresponding to the physical cultural relic model.

[0087] The delivery module 20 is used to deliver one of the physical artifact models of different materials to the designated interactive area according to the progress of the content preview.

[0088] The fusion module 30 is used to identify the spatial information of the physical cultural relic model using spatial positioning technology, and to fuse the spatial information of the physical cultural relic model with the pose of the virtual camera in the VR glasses device.

[0089] The gesture tracking device 40 performs real-time gesture tracking and acquires real-time gesture tracking data.

[0090] The rendering module 50 is used to render a virtual hand in virtual space based on real-time gesture tracking data.

[0091] The interaction module 60 is used to establish a spatial mapping relationship between the virtual cultural relic model and the physical cultural relic model through spatial registration technology, so that the pose of the virtual cultural relic in the virtual space is synchronized with the pose of the physical cultural relic in the real space in real time. When the audience wears VR glasses to observe the virtual scene, while their real hand touches the physical cultural relic model in the designated interactive area, their virtual hand simultaneously touches the corresponding virtual cultural relic model in the virtual space.

[0092] This application also provides a computer storage medium storing computer instructions, which, when invoked, execute the address mapping method of the large-capacity solid-state drive. The computer storage medium includes one or more program instructions, which are executed by a processor to provide a virtual tactile interaction method for cultural relics based on multimodal perception fusion.

[0093] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions. When the computer program instructions are executed on a computer, the computer executes the above-described method for virtual tactile interaction of cultural relics based on multimodal perception fusion.

[0094] This invention provides a processor for processing the above-described method for virtual tactile interaction of cultural relics based on multimodal perception fusion.

[0095] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0096] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0097] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0098] The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EEPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0099] The beneficial effects achieved by this application are as follows: (1) This application transmits the physical cultural relic model to the designated interactive area in front of the audience, ensuring that the physical anchor point of the physical cultural relic model coincides with the spatial positioning reference point of the designated interactive area, and that the posture of the physical cultural relic model remains stable.

[0100] (2) This application eliminates the fusion error between the three-dimensional spatial pose data of the physical cultural relic model and the real-time pose data of the virtual camera by using an error compensation algorithm, thereby improving the accuracy of virtual-real pose fusion.

[0101] (3) This application scores the confidence of multiple gesture acquisition devices, and optimizes the data collected by the gesture acquisition devices based on the confidence scores to improve the accuracy of the data collected by the gesture acquisition devices.

[0102] (4) This application maps the collected hand key point tracking data to a pre-made three-dimensional virtual hand model to achieve real-time synchronization of the virtual hand position, posture, finger flexion and extension movements with the audience's real gestures without delay.

[0103] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0104] In the description of this application, the word "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0105] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A virtual tactile interaction method for cultural relics based on multimodal perception fusion, characterized in that, The method includes: Preprocessing of 3D ancient scenes and 3D digital models of cultural relics corresponding to physical cultural relic models; Based on the progress of the content preview, one of the physical artifact models of different materials will be sent to the designated interactive area; Spatial positioning technology is used to identify the spatial information of physical cultural relic models, and the spatial information of physical cultural relic models is fused with the pose of virtual cameras in VR glasses devices; Start the gesture tracking device to perform real-time gesture tracking, acquire real-time gesture tracking data, and render a virtual hand in virtual space based on the real-time gesture tracking data. Spatial registration technology is used to establish a spatial mapping relationship between virtual and physical artifact models, so that the pose of virtual artifacts in virtual space is synchronized with the pose of physical artifacts in real space in real time. When viewers wear VR glasses to observe the virtual scene, their real hands touch the physical artifact model in the designated interactive area, while their virtual hands simultaneously touch the corresponding virtual artifact model in the virtual space.

2. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 1, characterized in that, The preprocessing of the 3D ancient scene and 3D digital model of the artifact, corresponding to the physical artifact model, includes: Collect attribute and characteristic information of physical cultural relic models; Based on the historical background and excavation environment of the physical artifact model, the corresponding three-dimensional ancient scene is called up through modeling software; Based on the material of the physical artifact model, a three-dimensional digital model of the artifact is generated using modeling software, matching the scale of the physical artifact model.

3. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 2, characterized in that, The attribute characteristics of physical cultural relic models include historical background, excavation environment, and materials.

4. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 1, characterized in that, Depending on the progress of the content preview, one of the physical artifact models of different materials will be sent to the designated interactive area, including: Initiate the content preview program for cultural relic interaction. Through the progress node markers within the content preview program, identify the current progress stage of the content preview in real time and determine the type of cultural relic material to be displayed at the current progress stage of the content preview. Based on the determined material type of the cultural relic, retrieve the corresponding three-dimensional digital model of the cultural relic from the physical cultural relic model storage area; The physical artifact model is transported to the designated interactive area directly in front of the audience using a mechanical conveyor system.

5. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 1, characterized in that, The process involves activating the gesture tracking device to perform real-time gesture tracking, acquiring real-time gesture tracking data, and rendering a virtual hand in virtual space based on that data. Activate the gesture tracking device and determine the spatial range for gesture acquisition; Based on the spatial range of gesture acquisition, real-time gesture tracking is performed, and real-time gesture tracking data is collected. Real-time gesture tracking data is mapped onto a 3D virtual hand model to generate a virtual hand in virtual space.

6. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 5, characterized in that, Based on the spatial range of gesture acquisition, real-time gesture tracking is performed. After acquiring real-time gesture tracking data, the process also includes: confidence scoring of multiple gesture acquisition devices, and optimization of the data acquired by the gesture acquisition devices based on the confidence scores.

7. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 6, characterized in that, Confidence scores were assigned to multiple gesture recognition devices, and the data collected by these devices was optimized based on these confidence scores, including: Acquire confidence evaluation index data for gesture acquisition devices; Based on the measurement variance of the gesture acquisition device, calculate the confidence score of the gesture acquisition device, and generate a standardized data stream carrying the confidence score of the gesture acquisition device; The data in the standardized data stream is optimized based on the confidence score of the gesture acquisition device.

8. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 5, characterized in that, After generating the virtual hand, the rendering of the virtual hand is optimized.

9. The virtual tactile interaction method for cultural relics based on multimodal perception fusion according to claim 1, characterized in that, Spatial registration technology is used to establish a spatial mapping relationship between virtual and physical artifact models, enabling real-time synchronization between the virtual artifact's pose in virtual space and the physical artifact's pose in real space. When a viewer wears VR glasses to observe a virtual scene, while their real hand touches the physical artifact model in a designated interactive area, their virtual hand simultaneously touches the corresponding virtual artifact model in virtual space, including: Establish a spatial mapping relationship between virtual cultural relic models and physical cultural relic models to ensure that the position, posture, and size of virtual cultural relics in VR space are synchronized with the actual posture of physical cultural relics in physical space in real time. Real-time detection of the physical contact between the viewer's real hand and the physical artifact model, and synchronous triggering of virtual hand-to-art touch events between the virtual hand and the virtual artifact model; Based on the location and type of physical touch, multimodal interactive feedback is rendered in the VR virtual space to enhance the unified experience of vision and touch.

10. A virtual tactile interaction system for cultural relics based on multimodal perception fusion, characterized in that, The system for performing the method according to any one of claims 1-9 comprises: The preprocessing module is used to preprocess the 3D ancient scene and 3D digital model of the cultural relic corresponding to the physical cultural relic model; The delivery module is used to deliver one of the physical artifact models of different materials to the designated interactive area according to the progress of the content preview. The fusion module is used to identify the spatial information of the physical cultural relic model using spatial positioning technology, and then fuse the spatial information of the physical cultural relic model with the pose of the virtual camera in the VR glasses device. Gesture tracking devices perform real-time gesture tracking and acquire real-time gesture tracking data; The rendering module is used to render a virtual hand in virtual space based on real-time gesture tracking data; The interaction module is used to establish a spatial mapping relationship between virtual cultural relic models and physical cultural relic models through spatial registration technology, so that the pose of virtual cultural relics in virtual space is synchronized with the pose of physical cultural relics in real space in real time. When the audience wears VR glasses to observe the virtual scene, while their real hand touches the physical cultural relic model in the designated interactive area, their virtual hand simultaneously touches the corresponding virtual cultural relic model in the virtual space.

Images