An NFC tactile interaction display method and system supporting real-time query of building material process

Abstract

The embodiment of the application relates to an NFC tactile interaction display method and system supporting real-time query of building material technology, and belongs to the technical field of electronic data processing. The method steps comprise the following: in response to a proximity event of a mobile terminal and an NFC tag, reading building material identification data and obtaining corresponding tactile waveform files, building material virtual model data and audio and video content from a server, establishing a master clock reference based on a zero buffer timestamp of an audio system, mapping technology node time in the tactile waveform to the master clock reference, generating a node trigger queue, comparing a current clock with a time difference value of the trigger queue in real time through an audio clock callback, producing synchronous touch, calculating a spatial position and an attitude matrix of a virtual model, superimposing the model into a real-time video stream to realize augmented reality display. Through multimodal data fusion and high-precision time sequence synchronization, the embodiment of the application realizes immersive building material technology display experience of audio-visual tactile integration.

Description

Technical Field

[0001] This application relates to an NFC tactile interactive display method and system that supports real-time query of building material processes, belonging to the technical field of electronic data processing. Background Technology

[0002] With the rapid development of the building decoration industry, consumers have placed higher demands on the purchasing experience of building materials. Existing building material display methods mainly rely on physical catalogs and model rooms, which have inherent shortcomings such as a single display format, limited information capacity, and inability to intuitively demonstrate construction techniques and final effects.

[0003] In recent years, with the advancement of mobile internet and augmented reality (AR) technologies, some mobile-terminal-based building material display solutions have emerged. For example, these solutions use QR codes or ordinary NFC tags as information carriers. By scanning electronic tags (such as NFC or RFID tags) on physical building materials, product information is obtained, and the corresponding virtual model's decorative effect in a house layout is displayed on the mobile device. While this NFC or RFID tag-based method achieves the conversion from physical objects to virtual models, its interaction remains at the visual level, exhibiting significant limitations. It lacks a demonstration of the building material production process, preventing users from understanding the manufacturing process and quality control procedures. Furthermore, the interaction is simplistic, providing only visual feedback and failing to simulate the realistic texture of the building materials, resulting in insufficient immersion for the user experience.

[0004] In summary, the existing technologies have the following problems that urgently need to be addressed: (1) the building material display methods are monotonous and lack an immersive experience that integrates multiple senses; (2) the synchronization accuracy between tactile feedback and multimedia content is insufficient, affecting the authenticity of the display; (3) the system resource scheduling efficiency is low when multiple tasks are processed concurrently, affecting the continuity of the user experience; and (4) there is a lack of intuitive display and interactive experience of the building material production process. Therefore, there is an urgent need for a building material process display solution that can integrate visual, auditory, and tactile multimodal interaction and ensure high-precision synchronization. Existing technologies can no longer meet people's requirements and urgently need to be improved. Summary of the Invention

[0005] The main objective of this application is to provide an NFC tactile interactive display method and system that supports real-time query of building material processes, thereby addressing the shortcomings of existing technologies.

[0006] The embodiments of this application adopt the following technical solutions: According to one aspect of the embodiments of this application, an NFC haptic interactive display method supporting real-time query of building material processes is provided, comprising: responding to a proximity event between a mobile terminal and an NFC tag for a building material process sample, reading building material identification data stored in the NFC tag, obtaining corresponding building material process sample information from a server based on the building material identification data, wherein the building material process sample information includes a haptic waveform file and building material virtual model data, loading building material process audio and video content from the building material process sample information onto the mobile terminal; before playing the building material process audio and video content on the mobile terminal, calling the audio clock interface of the mobile terminal and obtaining a zero-buffered audio timestamp corresponding to the actual sound output time of the speaker, and building a... A master clock reference is established, and the preset process node time values ​​in the tactile waveform file are mapped to the master clock reference to generate a node trigger queue. After playback starts, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through the registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the tactile feedback drive unit to make the vibration module of the mobile terminal generate tactile vibration. And / or: the image acquisition device of the mobile terminal captures the real environment in which the user is located, calculates the embedding position and posture transformation matrix of the building material virtual model in the real space, superimposes the virtual model onto the real-time image stream according to the transformation matrix, and performs interactive output on the display screen of the mobile terminal.

[0007] According to at least one specific embodiment of the present application, the step of loading the building material process audio and video content in the building material process sample information on the mobile terminal further includes: the mobile terminal initiating a query request to the server containing the building material identification data, wherein the query request encapsulates the device configuration information of the mobile terminal; the server receiving the query request, performing a query operation in one or more building material information databases, and retrieving the associated benchmark building material process sample information based on the building material identification data; the server performing adaptive processing on the benchmark building material process sample information according to the received device configuration information, wherein the adaptive processing specifically includes: generating a compatible haptic waveform file based on the device haptic feedback capability of the mobile terminal, obtaining the device graphics rendering capability of the mobile terminal, selecting the corresponding building material virtual model data from the multi-level detail model library, and selecting a simplified model and using compressed texture for devices with low rendering capability; the server encapsulating the building material process sample information after the adaptive processing into a response data packet and transmitting it back to the mobile terminal through the network.

[0008] According to at least one specific embodiment of the present application, the device configuration information includes: device model, operating system version, haptic feedback protocol version, graphics rendering capability level, and device haptic feedback capability; the building material process sample information includes haptic waveform files, building material virtual model data, and multimedia resource positioning information; the server receives a request uploaded by the mobile terminal, the request containing device configuration information; the server adaptively processes the building material process sample information according to the device configuration information, and encapsulates the processed virtual building material model data and an environmental 3D point cloud map into a response data packet; the mobile terminal receives and parses the response data packet, loads the environmental 3D point cloud map, and... The system identifies a physical plane as a reference, establishes a world coordinate system aligned with the physical plane, and determines the virtual building material model within this world coordinate system, thus completing the spatial registration of the virtual building material model. The mobile terminal acquires a video stream containing real-time building material process sample information and uses a visual tracking algorithm to solve for the six-degree-of-freedom pose of the camera relative to the loaded 3D point cloud map of the environment. Using the six-degree-of-freedom pose, combined with camera intrinsic parameters, the three-dimensional vertices of the spatially registered virtual model are transformed by perspective projection to calculate their two-dimensional coordinates on the current image plane. The virtual model is then rendered in real-time to generate a virtual image layer, which is then superimposed and synthesized with the real-time video stream to output the superimposed augmented reality image.

[0009] According to at least one specific embodiment of the present application, before playing building material process audio-visual content on the mobile terminal, the step of calling the audio clock interface of the mobile terminal and obtaining a zero-buffered audio timestamp corresponding to the actual sound output time of the speaker, establishing a master clock reference based on the zero-buffered audio timestamp, mapping the preset process node time values ​​in the tactile waveform file to the master clock reference, and generating a node trigger queue, further includes: when starting the playback of building material process audio-visual content, submitting the first frame audio data block to the audio hardware interface, and obtaining the expected presentation timestamp of the first frame audio data block returned by the audio system of the mobile terminal. Based on the expected presentation timestamp, a master clock reference synchronized with the audio hardware is established; the tactile waveform file in the building material process sample information is parsed, the process node time parameters in the tactile waveform file are extracted, and the node time values ​​are converted to the master clock reference coordinate system through a time mapping algorithm to generate a process node trigger queue based on time sequence arrangement; during playback, the playback progress is monitored in real time through an audio clock callback mechanism, and when the time difference between the current playback time and the target time in the process node trigger queue is less than a set threshold, a trigger command is sent to the tactile driving unit to realize the synchronization of audio playback and tactile feedback of building material process content.

[0010] According to at least one specific embodiment of the present application, the step of parsing the tactile waveform file in the building material process sample information, extracting the process node time parameters in the tactile waveform file, converting the node time values ​​to the master clock reference coordinate system through a time mapping algorithm, and generating a process node trigger queue based on time sequence arrangement further includes: parsing the metadata segment of the tactile waveform file, extracting the process node time parameters in the metadata segment with the media time axis as the reference system, and obtaining the relative time offset of each process node; establishing a system-level master clock reference based on the zero-buffer timestamp obtained from the audio system of the mobile terminal, converting the relative time offset into an absolute time coordinate based on the master clock using a time mapping function; calculating the trigger time of all nodes in the absolute time coordinate system by performing an addition operation on the relative time offset of each process node and the zero-buffer timestamp, and generating a sorted process node trigger queue according to the chronological order.

[0011] According to at least one specific embodiment of the present application, after playback is started, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through a registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback driving unit to cause the vibration module of the mobile terminal to generate haptic vibration. This further includes: after the media player on the mobile terminal is started, registering a high-precision clock callback function with the audio system of the mobile terminal to establish a real-time monitoring channel based on the hardware audio clock; periodically obtaining the sampled value of the current audio clock counter through the callback function, and reading the target time of the next node to be triggered from the node trigger queue, calculating the time difference between the current clock value and the target time; when the time difference is detected to be less than a preset trigger threshold, sending a trigger command containing waveform parameters to the haptic feedback driving unit to drive the vibration module to execute the corresponding haptic effect.

[0012] According to at least one specific embodiment of the present application, the step of registering a high-precision clock callback function with the audio system of the mobile terminal after the media player on the mobile terminal is started, and establishing a real-time monitoring channel based on the hardware audio clock, further includes: after the media player completes initialization, registering a high-precision clock callback function based on hardware interrupt driver by calling the audio service interface at the underlying level of the operating system; configuring the execution parameters of the callback function, including setting the sampling interval time, specifying the priority level, and allocating a dedicated interrupt handling thread; and each time the callback function is called, obtaining the corresponding system audio time by reading the current value of the audio hardware clock counter, and sending the system audio time to the trigger determination module, which is used to determine whether the mobile terminal needs to be triggered to vibrate at the current moment.

[0013] According to at least one specific embodiment of the present application, the image acquisition device of the mobile terminal captures the real environment in which the user is located, calculates the embedding position and attitude transformation matrix of the virtual building material model in the real space, superimposes the virtual model onto the real-time image stream according to the transformation matrix, and outputs it interactively on the display screen of the mobile terminal. The method further includes: starting the image acquisition device of the mobile terminal to capture the real-world environment image stream; initializing the SLAM engine; tracking the six-degree-of-freedom pose of the mobile terminal in physical space in real time by fusing data from the camera and inertial measurement unit; based on the position command input by the user, converting the two-dimensional screen coordinates into three-dimensional spatial coordinates using a ray casting algorithm; obtaining the embedding position of the virtual model in the real-world environment through spatial geometric calculations; and calculating the attitude transformation matrix that transforms the model from the local coordinate system to the world coordinate system based on the embedding position; inputting the attitude transformation matrix and the virtual building material model data into the graphics rendering engine; implementing the spatial transformation of the model through vertex shaders; completing material lighting calculations using fragment shaders; and compositing the rendered virtual model with the real-time environment image stream through alpha blending; and outputting augmented reality images on the display screen of the mobile terminal.

[0014] According to at least one specific embodiment of the present application, the step of inputting the pose transformation matrix and building material virtual model data into a graphics rendering engine, realizing spatial transformation of the model through vertex shaders, completing material lighting calculations using fragment shaders, compositing the rendered virtual model with the real-time environment image stream through alpha blending, and outputting augmented reality images on a mobile terminal display screen further includes: receiving touch input events from the user on the mobile terminal touchscreen, obtaining two-dimensional coordinate data of the touch point in the screen coordinate system, and normalizing the two-dimensional coordinate data to the projection coordinate system; based on the current camera parameters of the mobile terminal and the camera pose matrix provided by the SLAM system, converting the normalized screen coordinates into a three-dimensional spatial ray emanating from the camera focus through back projection calculation; using a spatial geometric intersection algorithm, calculating the intersection point of the three-dimensional spatial ray with the three-dimensional mesh model of the environment constructed by the SLAM system, and outputting the three-dimensional spatial coordinates of the first valid intersection point as the transformation result.

[0015] According to another aspect of the embodiments of this application, an NFC haptic interactive display system supporting real-time query of building material processes is provided, used to implement the NFC haptic interactive display method supporting real-time query of building material processes, including: a building material process audio-visual loading module, which, in response to a proximity event between a mobile terminal and an NFC tag for a building material process sample, reads building material identification data stored in the NFC tag, obtains corresponding building material process sample information from a server based on the building material identification data, the building material process sample information including a haptic waveform file and building material virtual model data, and loads the building material process audio-visual content from the building material process sample information onto the mobile terminal; and a haptic waveform file node triggering generation module, which, before the mobile terminal plays the building material process audio-visual content, calls the audio clock interface of the mobile terminal and obtains a zero-buffered sound corresponding to the actual sound output time of the speaker. The system includes a frequency timestamp, a master clock reference established based on the zero-buffered audio timestamp, mapping the preset process node time values ​​in the tactile waveform file to the master clock reference, and generating a node trigger queue; a mobile terminal vibration control module, after playback starts, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through a registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the tactile feedback drive unit to cause the vibration module of the mobile terminal to generate tactile vibration; and / or: a mobile terminal interactive output module, the mobile terminal's image acquisition device captures the user's real environment, calculates the embedding position and posture transformation matrix of the building material virtual model in the real space, superimposes the virtual model onto the real-time image stream according to the transformation matrix, and performs interactive output on the mobile terminal's display screen.

[0016] The beneficial technical effects of the embodiments of this application are: This application adopts a multimodal data fusion and timing synchronization approach. By coordinating four technical links—NFC near-field triggering, audio clock reference establishment, tactile waveform synchronous mapping, and augmented reality spatial positioning—it achieves the goal of transforming traditional static building material displays into a dynamic, immersive, multi-sensory experience.

[0017] This application embodiment achieves the technical effect of haptic feedback and audio-visual content synchronization by establishing a mapping relationship between the master clock reference based on zero-buffered audio timestamps and haptic waveform nodes. This allows users to simultaneously experience a haptic experience that matches the video content while watching building material process videos, achieving audio-visual and tactile synchronization and creating an immersive experience.

[0018] This application embodiment achieves smooth collaborative operation of multimodal interaction by parallel processing of the haptic feedback channel and the augmented reality rendering channel, and by using unified device configuration information for resource scheduling. This allows users to experience synchronous haptic feedback while observing the display effect of virtual building materials in the real environment in real time. Through the reasonable allocation of mobile terminal and server resources, a panoramic interactive experience is obtained.

[0019] This application embodiment achieves the technical effect of intelligent building material display by using NFC triggering, data interaction between mobile terminal and server, local rendering and real-time interaction with haptic feedback. Users can obtain the display effect of building material process by interactive touch of mobile terminal and enjoy a full range of experience services including texture experience and scene preview. Attached Figure Description

[0020] To more clearly illustrate the specific implementation methods of the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the specific implementation methods or the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the NFC tactile interactive display method that supports real-time query of building material processes, provided in steps S1 to S4.

[0022] Figure 2 This is a flowchart of the optimization technical solutions provided in steps S11 to S13.

[0023] Figure 3 This is a flowchart of the optimization technical solutions provided in steps S21 to S23.

[0024] Figure 4 This is a flowchart of the optimization technical solutions provided in steps S31 to S33.

[0025] Figure 5 This is a flowchart of the optimization technical solutions provided in steps S41 to S43.

[0026] Figure 6 This is an architecture diagram of an NFC tactile interactive display system that supports real-time query of building material processes.

[0027] Figure 7 This is a schematic diagram of the electronic device. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the specific implementation methods in the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0029] The technical solutions of this application have wide applications in multiple application scenarios. The following is a detailed description of the implementation of this application in a real application scenario, taking the application in marble building materials display as an example: A user is in the marble exhibition area of ​​a building materials market and is interested in a marble slab called ****.

[0030] When a user brings their mobile device (smartphone) close to an NFC tag affixed to the display board, the mobile device automatically senses the proximity and retrieves the building material identification data "Marble_White_001" stored in the tag via the NFC reader module. Subsequently, the mobile device encapsulates the building material identification data and its phone configuration information (including device model, operating system version, haptic feedback protocol version, graphics rendering capability level, etc.) into a data request packet via the network module and sends it to the server.

[0031] After receiving the request, the server retrieves the corresponding process sample information from the building materials process database based on the identifier data (Marble_White_001). The retrieved information includes: (1) a 4K process video showing the entire process of the marble from mining to surface polishing; (2) tactile waveform files specifically designed for the mining, cutting, and polishing processes of this brand of marble; and (3) high-precision three-dimensional virtual model data of the marble. After adaptively optimizing the original data according to the configuration information of the mobile terminal, the server returns the processed data packet to the user's mobile phone.

[0032] Before the mobile terminal starts playing the process video, it first performs haptic synchronization preparation. By calling the AAudio interface of the Android system or the Core Audio interface of the iOS system, it obtains the zero-buffered audio timestamp directly associated with the speaker hardware buffer. Based on the zero-buffered audio timestamp, a master clock reference is established and the corresponding haptic waveform file is parsed. The preset process node time parameters in the haptic waveform file are extracted, such as: the Xth second corresponds to the mining impact, the Xth second corresponds to the cutting vibration, and the Xth second corresponds to the polishing fineness. Using the above time parameters, the relative time values ​​are converted to the master clock reference through a time mapping algorithm to generate the corresponding node trigger queue.

[0033] When the user clicks the play button, the mobile terminal activates two parallel processing channels: a haptic feedback channel and an augmented reality channel.

[0034] Haptic Feedback Channel: The mobile terminal registers a high-priority audio clock callback function to monitor the audio playback progress in real time. When the cutting process scene at the Xth second is played, the callback function detects that the difference between the current audio clock and the target time of the cutting process node is less than a set threshold of 5 milliseconds, and immediately sends a trigger signal to the haptic feedback drive unit. Upon receiving the trigger signal, the drive unit controls the linear resonant motor to generate short and strong vibrations, simulating the impact of the cutting machine contacting the marble surface.

[0035] Augmented Reality Channel: The phone's rear camera activates, capturing the actual environment in front of the user. SLAM technology is used to calculate the spatial plane of the display area, and a ray casting algorithm is employed to determine the optimal placement and orientation of the virtual marble slabs. The graphics rendering engine then overlays the 3D model of the virtual marble slabs onto the real-world image captured by the camera in real time, according to the calculated transformation matrix. Users can also observe the virtual marble's paving effect in real space from different angles using their mobile devices and interact with different slab sizes via gestures. During the experience, if a user is particularly interested in a particular process step, they can pause the video, and the mobile device will display detailed parameters for that step. When a user chooses to purchase, they can complete the transaction directly through the shopping cart button on the interactive interface. The mobile device and service also record the user's preferred slab size and virtual paving effect.

[0036] The main application scenarios of the above-described embodiments of this application demonstrate the complete process of the technical solution in practical applications. Through the collaborative work of multiple technical links such as NFC triggering, multimodal data acquisition, high-precision tactile synchronization and AR visualization, it provides users with an immersive and interactive building material process display experience, effectively solving the problems of insufficient information and monotonous experience in the existing technology.

[0037] like Figure 1 The NFC haptic interactive display method shown, which supports real-time query of building material processes, includes: Step S1: In response to the proximity event between the mobile terminal and the NFC tag of the building material process sample, read the building material identification data stored in the NFC tag, obtain the corresponding building material process sample information from the server according to the building material identification data, the building material process sample information includes tactile waveform files and building material virtual model data, and load the building material process audio and video content in the building material process sample information on the mobile terminal.

[0038] Step S2: Before playing the audio and video content of building material process on the mobile terminal, call the audio clock interface of the mobile terminal and obtain the zero-buffered audio timestamp corresponding to the actual sound output time of the speaker. Based on the zero-buffered audio timestamp, establish a master clock reference, map the preset process node time value in the tactile waveform file to the master clock reference, and generate a node trigger queue.

[0039] Step S3: After playback starts, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through the registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback drive unit to cause the vibration module of the mobile terminal to generate haptic vibration, and / or: Step S4: The image acquisition device of the mobile terminal captures the real environment in which the user is located, calculates the embedding position and posture transformation matrix of the virtual building material model in the real space, superimposes the virtual model onto the real-time image stream according to the transformation matrix, and performs interactive output on the display screen of the mobile terminal.

[0040] The technical solutions provided in steps S1 to S4 support real-time query of building material processes and enable tactile interactive display based on NFC. By constructing control logic for multimodal data collaborative processing and timing control, an interactive process from information triggering to multi-sensory presentation is realized. The interaction starts with the trigger response of NFC near-field communication. By establishing an audio hardware-level time reference, precise synchronization of touch is achieved. Spatial fusion calculation of augmented reality (AR) is executed in parallel, forming an immersive building material display experience integrating audiovisual and tactile senses for mobile terminal users.

[0041] Step S1 obtains complete process sample information containing tactile waveforms and virtual models, providing a data foundation for subsequent tactile synchronization and augmented reality (AR) display. Steps S2 and S3 adopt a synchronization control method based on audio hardware clock. Step S2 establishes a master clock reference aligned with the speaker's sound output time and generates a corresponding trigger queue. Then, the callback function in step S3 performs real-time comparison and triggering, achieving millisecond-level precise synchronization between tactile events and audio / video playback. This achieves the technical effect of deep integration of tactile feedback with building material process images and sound, preventing the problem of multi-sensory experience misalignment and distortion caused by timing errors at the software level.

[0042] As can be seen, in the technical solutions provided in steps S1 to S4, step S1 is used to obtain the data source, step S2 establishes the master clock reference and generates the node trigger queue, step S3 is responsible for processing time synchronization haptic feedback, and step S4 is responsible for processing spatial overlay rendering. This achieves the goal of efficient allocation of computing resources and seamless integration of user experience, enabling users to experience real-time haptic feedback while also smoothly interacting and observing the effects of virtual building materials in the real environment.

[0043] like Figure 2 As shown, preferably, in step S1, loading the building material process audio and video content from the building material process sample information onto the mobile terminal further includes: the mobile terminal initiating a query request to the server containing the building material identification data, wherein the query request encapsulates the mobile terminal's device configuration information, further including: Step S11: The server receives the query request, performs a query operation in one or more building material information databases, and retrieves the associated benchmark building material process sample information based on the building material identification data.

[0044] In step S12, the server adaptively processes the baseline building material process sample information based on the received device configuration information. Specifically, this adaptive processing involves: generating a compatible haptic waveform file based on the mobile terminal's haptic feedback capability; acquiring the mobile terminal's graphics rendering capability; selecting corresponding building material virtual model data from a multi-level detail model library; and for devices with low rendering capabilities, selecting a simplified model and using compressed textures. In step S12, a compatible haptic waveform file is generated using waveform matching or parameter scaling algorithms. The building material virtual model data corresponds to the device's graphics rendering capability, and the device configuration information is used to characterize the device's capabilities.

[0045] In step S13, the server encapsulates the building material process sample information after the adaptive processing into a response data packet and transmits it back to the mobile terminal via the network.

[0046] The optimization technical solutions provided in steps S11 to S13 construct a control method process for server adaptive data processing based on device capability awareness. Through two-way information interaction between the mobile terminal and the server, the multimedia content of building material process information is matched with the hardware performance of the mobile terminal and intelligent interaction is realized.

[0047] The main content of step S1 is to use the mobile terminal to actively report device configuration information and the server to intelligently adapt based on the device configuration information. The configuration information representing the device capabilities is encapsulated in the request, providing the server with the basis for adaptation. Steps S11 and S12 then perform data retrieval and adaptive processing based on this adaptation basis, achieving the goal of matching and aligning cloud resources with terminal hardware capabilities. This enables high-end devices to enjoy high-quality content, while ensuring a smooth experience for mid-range or low-end devices.

[0048] Step S12 adopts a parallel processing adaptation strategy for both tactile and graphical data. It generates compatible tactile waveform files based on the device's tactile feedback capabilities and selects model data with corresponding detail levels according to the device's graphics rendering capabilities. It optimizes the capabilities of different dimensions of mobile terminal devices to achieve the goal of on-demand resource allocation. Under the condition of limited network bandwidth and terminal computing power, it prioritizes core experiences such as basic tactile feedback and model display. After achieving the priority guarantee, it intelligently adjusts non-core experiences (such as model accuracy and texture details), thus solving the bottleneck problem of mobile terminal data processing.

[0049] In the coordinated operation of steps S12 and S13, a strategy of unified packaging and distribution after the server completes all adaptation work is adopted. In step S12, a compatible haptic waveform file is generated, and the virtual model data of building materials is optimized and selected. Then, in step S13, it is uniformly packaged, which reduces the computing pressure on the mobile terminal and optimizes the response time. This enables the mobile terminal to quickly receive and directly use the data, greatly shortening the delay from response triggering to screen display interaction.

[0050] As an example, in step S12, the device configuration information includes: device model, operating system version, haptic feedback protocol version, graphics rendering capability level, and device haptic feedback capability. The building material process sample information includes haptic waveform files, building material virtual model data, and multimedia resource positioning information. Although step S12 can output a virtual augmented reality model, in practice, it has been found that the augmented reality model generated in step S12 has defects such as instability and inaccuracy, which will affect the subsequent user experience. To solve the above problems, this application embodiment also provides further improvements to step S12:

[0051] In step S121, the server receives a request uploaded by the mobile terminal. The request includes device configuration information. The server adaptively processes the building material process sample information based on the device configuration information and encapsulates the processed virtual building material model data and an environmental 3D point cloud map into a response data packet. The server stores the environmental 3D point cloud map, which is obtained through high-precision 3D scanning and precisely corresponds to the user's physical environment.

[0052] Step S122: The mobile terminal receives and parses the response data packet, loads the environmental 3D point cloud map, identifies a physical plane as a reference in the loaded environmental 3D point cloud map, establishes a world coordinate system aligned with the physical plane, and determines the virtual building material model in the world coordinate system to complete the spatial registration of the virtual building material model. For example, the target position, size, and orientation of the virtual building material model are determined to complete the spatial registration. Spatial registration refers to the process of placing a virtual object in a specific location in real 3D space in an augmented reality system. This placement involves determining the position, orientation, and size of the virtual object and ensuring that the virtual object remains fixed in place like a real object when the smart terminal moves.

[0053] Step S123: The mobile terminal acquires a video stream of real-time building material process sample information and solves the six-degree-of-freedom pose of the camera relative to the loaded 3D point cloud map of the environment using a visual tracking algorithm.

[0054] Step S124: Using the six-degree-of-freedom pose and combined with the camera intrinsic parameters, the three-dimensional vertices of the virtual model that has completed spatial registration are used to calculate their two-dimensional coordinates on the current image plane through perspective projection transformation.

[0055] Step S125: The virtual model is rendered in real time to generate a virtual image layer. This virtual image layer is then overlaid and synthesized with the real-time video stream to output the composite augmented reality image. In this step, the augmented reality image has correct relationships between reality and illusion, as well as occlusion, and corresponds to the object's position in the environmental space.

[0056] The optimized technical solutions provided in steps S121 to S125 move the high-precision environment reconstruction task to the cloud, allowing the mobile terminal to focus on real-time tracking and rendering, thereby achieving millimeter-level accurate registration of virtual objects in physical space and seamless visual integration.

[0057] Steps S121 and S122 employ a rapid spatial registration method based on a prior environment model. In step S121, the server distributes adapted virtual model data and provides an environmental point cloud map obtained through 3D scanning. Step S122 utilizes the prior environmental knowledge (device configuration information) from step S121 to quickly identify the physical plane and establish the corresponding world coordinate system, skipping the terminal SLAM mapping stage and directly entering a stable tracking state. This achieves real-time and accurate positioning of the virtual model in the user environment of the mobile terminal, with centimeter-level registration accuracy. In existing technologies, real-time mapping on mobile terminals requires feature extraction and matching optimization, which is time-consuming and can easily cause a long delay in augmented reality (AR) startup (several seconds to tens of seconds), thus avoiding a poor user experience.

[0058] The world coordinate system established in step S122 provides a reference for the entire tracking process. Step S123, based on step S122, combines inertial measurement unit data and iteratively solves the six-degree-of-freedom pose of the camera through feature point matching and bundle adjustment optimization. Step S124 then uses the six-degree-of-freedom pose solved in step S123 to perform accurate perspective projection transformation, thereby achieving the goal of maintaining the persistent stability of the virtual model in physical space. Even if the mobile terminal moves over a wide range, rotates rapidly, or is partially occluded, the virtual model can still maintain a stable position with a small pose tracking error, avoiding the problem of significant virtual model drift after long-term use caused by accumulated errors.

[0059] In the coordinated operation of steps S124 and S125, a virtual-real fusion method based on depth testing is adopted. Step S124 accurately calculates the two-dimensional coordinates and depth information, and step S125 performs depth testing and stencil testing in the rendering pipeline to handle the relationship between the virtual model and the real environment. This achieves the goal of simulating the lighting and occlusion effects of the real world, enabling the virtual building material model to generate realistic shadows according to the ambient lighting and to be naturally occluded by real objects. This improves the naturalness of visual fusion and avoids the problem of virtual objects appearing to float, giving the final virtual objects a sense of three-dimensionality and realism.

[0060] like Figure 3 As shown, preferably, in step S2, before playing the audio-visual content of building material processes on the mobile terminal, the audio clock interface of the mobile terminal is called to obtain a zero-buffered audio timestamp corresponding to the actual sound output time of the speaker. A master clock reference is established based on the zero-buffered audio timestamp, and the preset process node time values ​​in the tactile waveform file are mapped to the master clock reference to generate a node trigger queue. This further includes: Step S21: When starting the playback of audio and video content of building material process, the first frame audio data block is submitted to the audio hardware interface, the expected presentation timestamp of the first frame audio data block returned by the audio system of the mobile terminal is obtained, and a master clock reference synchronized with the audio hardware is established based on the expected presentation timestamp.

[0061] Step S22: Parse the tactile waveform file in the building material process sample information, extract the process node time parameters from the tactile waveform file, and convert the node time values ​​to the master clock reference coordinate system using a time mapping algorithm to generate a process node trigger queue based on time sequence arrangement. The node time value here is one of the parameters in the process node time parameters.

[0062] Step S23: During playback, the playback progress is monitored in real time through an audio clock callback mechanism. When the time difference between the current playback time and the target time in the process node trigger queue is less than a set threshold, a trigger command is sent to the tactile driving unit to realize the synchronization of audio playback and tactile feedback of building material process content.

[0063] The optimized technical solutions provided in steps S21 to S23 construct a precise haptic synchronization control system based on audio hardware-level timing. By establishing a master clock reference strictly aligned with the audio output, precise synchronization between haptic feedback and audio / video content is achieved.

[0064] In the coordinated operation of steps S21 and S22, a master clock reference based on the expected presentation timestamp of the audio hardware is adopted. Step S21 obtains the expected presentation timestamp of the first frame data block returned by the audio system, using the first frame data block as the absolute time starting point for synchronization. Step S22 precisely maps the process node time parameters in the tactile waveform to this reference coordinate system, unifying the media time at the software level with the audio output time at the hardware level, ensuring that the tactile events and audio playback are strictly aligned from the start. In the coordinated operation of steps S22 and S23, a node queue based on time sequence arrangement and a real-time monitoring mechanism are adopted. The ordered trigger queue generated in step S22 provides the trigger target sequence for step S23. Step S23 continuously monitors the deviation between the current playback progress and the target time in the queue through high-frequency audio clock callbacks, dynamically managing the timing of multiple tactile events during playback. Multiple tactile nodes (such as continuous impacts and vibration changes) during the building material process are accurately reproduced, preventing the problem of timing chaos in complex tactile patterns caused by simple triggering based on playback progress percentage.

[0065] In summary, the hardware-level time reference established in step S21 ensures the time accuracy of the entire system. The mapping and transformation in step S22, based on step S21, guarantees the precision of the tactile event definition. Step S23 achieves the final precise execution through real-time comparison and threshold judgment. From step S21 to step S23, an end-to-end control system from content definition to hardware execution is constructed, enabling millisecond-level synchronization between tactile feedback and specific building material process actions in the audio and video content (such as the moment of cutting contact or the start of polishing). The synchronization error is controlled at the millisecond level, avoiding the significant lag or advance of tactile feedback caused by the inherent latency and scheduling uncertainty of the operating system's multimedia framework. This makes the interactive display of building material process information immersive.

[0066] As an example, in step S22, the step of parsing the tactile waveform file in the building material process sample information, extracting the process node time parameters from the tactile waveform file, converting the node time values ​​to the master clock reference coordinate system through a time mapping algorithm, and generating a process node trigger queue based on time sequence arrangement further includes: Step S221: Parse the metadata segment of the tactile waveform file, extract the process node time parameters in the metadata segment with the media time axis as the reference, and obtain the relative time offset of each process node.

[0067] Metadata segments are structured data blocks, typically embedded in the header or a specific index area of ​​a haptic waveform file in key-value pairs, XML, or JSON format. Their purpose is to provide timing and contextual information for haptic feedback. Located within the haptic waveform file, metadata segments describe and define when, where, and how haptic events are triggered. Metadata segments can define which key time points on the media timeline (i.e., time 0 relative to the start of audio / video content playback) require haptic feedback to be triggered. Example data:

[0068] "node_1": { "time_offset_ms": 3000, "type": "impact"} indicates that an impact-type haptic is triggered at the 3rd second of the video playback.

[0069] "node_2": { "time_offset_ms": 8500, "type": "grinding"} indicates that a grinding-type tactile sensation is triggered at 8.5 seconds.

[0070] node_1 and node_2, as time parameters, are the relative time offsets extracted in step S221, which form the basis for subsequent time mapping. As a further improvement to step S211, the relative time offsets and corresponding tactile intensity parameters of the first node (node_1) and the second node (node_2) can be extracted by parsing the metadata segment of the tactile waveform file, thereby obtaining the dependency information of the nodes and establishing a logical association model between the nodes.

[0071] After establishing the master clock reference based on the zero-buffer timestamp of the audio system, the deviation coefficient between the actual playback rate and the standard rate is calculated by the dynamic time warping algorithm. The relative time offset between the first node (node_1) and the second node (node_2) is adaptively corrected using the deviation coefficient to obtain the logical dependency relationship between the nodes. Based on the logical dependencies between the nodes, a trigger condition judgment mechanism based on a finite state machine is constructed. When the first node (node_1) is successfully triggered, the second node (node_2) is added to the queue to be triggered. The fault tolerance range of the trigger threshold is dynamically adjusted by real-time monitoring of system load, and a process node trigger queue with priority and dependency is generated.

[0072] The above improvement to step 221, by constructing a trigger condition judgment mechanism based on dependency, and adding the second node to the queue to be triggered only after the first node is successfully triggered, and dynamically adjusting the trigger threshold according to the system load, achieves the goal of optimizing the use of system resources while ensuring the correctness of haptic logic. It can still maintain the reliable triggering of key haptic sequences under high load, avoid haptic loss caused by system overload, and prevent the problem of decreased reliability of haptic feedback when system resources are tight due to the fixed threshold trigger mechanism.

[0073] Step S222: Based on the zero-buffered timestamp obtained from the audio system of the mobile terminal, a system-level master clock reference is established, and the relative time offset is converted into an absolute time coordinate based on the master clock using a time mapping function.

[0074] Step S223: By performing an addition operation between the relative time offset of each process node and the zero-buffer timestamp, the trigger time of all nodes in the absolute time coordinate system is calculated, and a sorted process node trigger queue is generated according to the chronological order.

[0075] In the optimization technical solutions provided in steps S221 to S223, a precise coordinate transformation system from media relative time to audio hardware absolute time is used to transform the description of tactile events into timing instructions that can be precisely executed by the hardware based on the digital mapping relationship.

[0076] The content porting and conversion in steps S221 and S222 adopts a mapping mechanism based on the relative offset of the media time axis and the absolute time reference of the audio hardware. Step S221 extracts the relative time parameters with media time as a reference from the haptic waveform file. Step S222 uses the zero-buffered timestamp obtained from the audio system to establish a system-level master clock reference and completes the coordinate system transformation through the time mapping function. A deterministic relationship is established between the relative time of the content definition and the absolute time of the hardware execution. This enables the decoupling of the haptic event definition and the hardware execution environment to achieve re-association. By controlling the consistency between the haptic event time description and the playback system time reference, problems such as difficulty in content porting and adaptation and difficulty in guaranteeing synchronization accuracy are avoided.

[0077] Steps S222 and S223 mainly involve collaborative computing and adopt a deterministic time coordinate transformation algorithm based on addition. The master clock reference established in step S222 provides a reference value for the addition operation in step S223. Step S223 calculates the absolute trigger time by performing an addition operation on the relative time offset of each node with this reference value. This achieves high-precision time coordinate transformation with relatively low computational complexity, enabling the rapid completion of timing calculations for large-scale haptic event queues on resource-constrained mobile devices while ensuring low computational latency.

[0078] In summary, the coordinated operation of steps S221, S222, and S223 employs control logic from data parsing and baseline establishment to queue generation. Step S221 ensures the accurate extraction of raw time data, step S222 provides a reliable reference clock, and step S223 completes timing calculations and queue sorting. Steps S221 to S223 construct a complete and reliable tactile timing preprocessing logic, enabling the generation of the tactile trigger queue and audio playback preparation to be completed synchronously. This significantly reduces system startup latency and avoids the problems of slow system startup and visual effect interruptions caused by the sequential execution of tactile timing calculations and audio playback initialization.

[0079] like Figure 4 As shown, preferably, in step S3, after playback is started, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through the registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback driving unit to cause the vibration module of the mobile terminal to generate haptic vibration, further including: Step S31: After the media player on the mobile terminal is started, a high-precision clock callback function is registered with the audio system of the mobile terminal to establish a real-time monitoring channel based on the hardware audio clock.

[0080] Step S32: Periodically obtain the sampled value of the current audio clock counter through the callback function, read the target time of the next node to be triggered from the node trigger queue, and calculate the time difference between the current clock value and the target time.

[0081] Step S33: When the time difference is detected to be less than the preset trigger threshold, a trigger command containing waveform parameters is sent to the tactile feedback drive unit to execute the corresponding tactile effect by driving the vibration module.

[0082] The optimization solutions provided in steps S31 to S33 achieve precise synchronization between audio playback and haptic feedback by establishing a high-precision real-time monitoring and triggering mechanism based on a hardware clock in the mobile terminal audio system. Steps S31 to S33, through the close coordination of three steps—registering a clock callback function, periodically calculating the time difference, and threshold triggering—use the audio clock as a time reference to precisely control the triggering timing of the haptic effect.

[0083] The monitoring channel established in step S31 provides a precise time reference for the entire system. By registering a high-precision clock callback function with the audio system, the hardware audio clock is directly used as the time source, fundamentally ensuring the accuracy of time measurement. The periodic calculation function in step S32 works in conjunction with step S31. The callback function periodically obtains the current audio clock value and calculates the difference with the target time in the node trigger queue. This continuous monitoring mechanism ensures that the system can track the playback progress in real time, providing data support for precise triggering. The threshold triggering mechanism in step S33 corresponds to the previous two steps. When the detected time difference is less than a preset threshold, the system immediately sends a trigger command, achieving millisecond-level response and ensuring that the haptic effect is accurately presented at a specific moment during audio playback.

[0084] In summary, step S31 provides a time reference, step S32 performs real-time monitoring and calculation, and step S33 executes the trigger. Through the coordinated action of steps S31 to S33, the audio and haptic feedback can achieve precise synchronization at the millisecond level, significantly improving the real-time performance and accuracy of the haptic effect during media playback, and bringing users a more immersive multimedia experience of building material process information.

[0085] As an example, in step S31, after the media player on the mobile terminal starts, registering a high-precision clock callback function with the audio system of the mobile terminal and establishing a real-time monitoring channel based on the hardware audio clock further includes: Step S311: After the media player completes initialization, a high-precision clock callback function based on hardware interrupt driver is registered by calling the underlying audio service interface of the operating system.

[0086] Step S312: Configure the execution parameters of the callback function, including setting the sampling interval, specifying the priority level, and allocating a dedicated interrupt handling thread to ensure that the callback function can be triggered and executed at a stable frequency.

[0087] Step S313: Each time the callback function is called, the corresponding system audio time is obtained by reading the current value of the audio hardware clock counter, and the system audio time is sent to the trigger determination module. The trigger determination module is used to determine whether the mobile terminal needs to be triggered to vibrate at the current moment.

[0088] The optimization solutions provided in steps S311 to S313 establish a hardware interrupt-driven clock sampling mechanism at the operating system level of the smart terminal, providing a time reference for triggering haptic feedback. Since the clock sampling mechanism uses a progressive design pattern across three levels—from underlying registration and parameter configuration to time transmission—it ensures that the system audio time is stably collected and delivered to the application layer modules.

[0089] Step S311 registers a callback function by calling the underlying audio service interface of the operating system and based on hardware interrupt drivers. This allows the time sampling operation to overcome the latency and jitter interference caused by software scheduling, providing a high-precision clock signal. This enables step S312 to accurately configure resources, build a stable execution environment for the smart terminal, and ensure that the callback function can be executed stably at the expected frequency, preventing other tasks within the operating system from preempting resources in the sampling process. Step S313 is responsible for sending the read precise hardware time value in real time and accurately each time a stable callback is triggered, providing a stable decision basis for response and triggering. In steps S311 to S313, the trigger determination module is a logical unit responsible for decision-making. Its function is to receive the system audio time, compare the system audio time with the preset haptic effect timeline, and determine whether the precise moment for triggering a certain haptic effect has been reached, and then issue an instruction to the haptic driver unit. The trigger determination module is usually implemented in the form of a software algorithm. It can be a component inside the media player of the mobile terminal, part of the multimedia framework of the operating system, or a separately developed middleware.

[0090] It can be seen that steps S311 to S313 work together to enable the triggering judgment module to obtain a continuous, accurate and low-jitter system audio time signal, thus providing a data processing foundation for the final realization of the synchronization of audio and tactile effects of building material process information.

[0091] For example, in an immersive experience of building material processes, the standardized installation process of double-glazed windows can be demonstrated using mobile devices (VR headsets, mobile phones, tablets, etc.). When the scene of applying sealant appears on the mobile device, in addition to playing audio, haptic feedback is also provided to simulate the resistance of applying the sealant. Specifically:

[0092] The mobile terminal will pre-generate a process tactile timeline that is precisely aligned with the audio and video tracks of the building materials process. In the video, the application of sealant occurs at 02:15.500 on the audio timeline, and the corresponding sealant application start tactile feedback (waveform ID: SEALANT_START) is also set to be triggered at 02:15.500 to simulate the resistance of sealant application. From 02:16.000 to 02:18.000, the corresponding sealant application resistance tactile feedback (waveform ID: SEALANT_RUN) also occurs during this period, and the vibration of the mobile terminal generates a feeling that simulates the resistance of sealant extrusion.

[0093] like Figure 5 As shown, preferably, in step S4, the image acquisition device of the mobile terminal captures the real environment in which the user is located, calculates the embedding position and pose transformation matrix of the virtual building material model in the real space, superimposes the virtual model onto the real-time image stream according to the transformation matrix, and performs interactive output on the display screen of the mobile terminal, further including: Step S41: Start the image acquisition device of the mobile terminal to capture the image stream of the real environment, initialize the SLAM engine, and track the six-degree-of-freedom pose of the mobile terminal in the physical space in real time by fusing the data from the camera and the inertial measurement unit.

[0094] Step S42: Based on the user's input position command, the two-dimensional screen coordinates are converted into three-dimensional spatial coordinates using a ray casting algorithm. The embedding position of the virtual model in the real environment is obtained through spatial geometric calculations. Based on the embedding position, the pose transformation matrix that transforms the model from the local coordinate system to the world coordinate system is calculated.

[0095] Step S43: Input the pose transformation matrix and building material virtual model data into the graphics rendering engine, realize the spatial transformation of the model through the vertex shader, complete the material lighting calculation through the fragment shader, and synthesize the rendered virtual model with the real-time environment image stream through Alpha blending, and output the augmented reality image on the mobile terminal display screen.

[0096] The optimized technical solution provided in steps S41 to S43 integrates environmental perception, spatial computation, and real-time rendering control logic. It embeds virtual building material models into the augmented reality effect of the user's real-time observed environment, displaying and interacting with them on the mobile terminal, thus constructing an AR interactive process from physical scene to virtual content presentation. Step S41 initializes the SLAM engine and fuses camera and IMU data to calculate the six-degree-of-freedom pose of the mobile terminal in physical space in real time. Its output pose data ensures that virtual objects do not drift or jitter with device movement. Step S42 transforms the user's two-dimensional interaction intent into anchor points in three-dimensional space and performs corresponding mapping. Through ray casting and spatial geometry calculations, it converts the user's position commands on the screen into specific coordinates and orientations in the real world. Then, a pose transformation matrix connects the digital virtual space and the physical reality, setting the positioning and orientation for subsequent rendering. Step S43 receives the transformation matrix (which determines the model's position and pose) and building material virtual model data (including geometry and materials) output from step S42. The transformation matrix determines the position and pose of the building material virtual model. Spatial transformation is completed through the vertex shader of the graphics rendering engine. Material and lighting calculations are then performed through the fragment shader. Finally, augmented reality (AR) functions are integrated into the real-time environment image stream through Alpha blending technology.

[0097] Through the control logic from steps S41 to S43, users can intuitively see how virtual building materials (such as flooring, tiles, and furniture) are placed in the actual environment with the correct scale, position, and lighting effects in a real building materials market, unfinished house, or design showroom via a mobile terminal screen. This greatly enhances the immersiveness and efficiency of building materials product display, interior design, and decision-making.

[0098] As an example, in step S43, the step of inputting the pose transformation matrix and the building material virtual model data into the graphics rendering engine, realizing the spatial transformation of the model through the vertex shader, completing the material lighting calculation using the fragment shader, compositing the rendered virtual model with the real-time environment image stream through alpha blending, and outputting the augmented reality image on the mobile terminal display screen further includes: Step S431: By receiving the user's touch input event on the mobile terminal touch screen, the two-dimensional coordinate data of the touch point in the screen coordinate system is obtained, and the two-dimensional coordinate data is normalized to the projection coordinate system.

[0099] Step S432: Based on the current camera parameters of the mobile terminal and the camera pose matrix provided by the SLAM system, the normalized screen coordinates are converted into three-dimensional spatial rays originating from the camera focus through back projection calculation.

[0100] Step S433: Using a spatial geometry intersection algorithm, calculate the intersection point between the three-dimensional spatial ray and the three-dimensional mesh model of the environment constructed by the SLAM system, and output the three-dimensional spatial coordinates of the first valid intersection point as the transformation result.

[0101] The optimized technical solutions provided in steps S431 to S433 realize the computational process from two-dimensional screen interaction to three-dimensional spatial positioning, mapping the user's touch points on the screen to their corresponding specific locations in the real three-dimensional environment, thus achieving a connection between the digital world and the physical world at the spatial interaction level. Step S431 captures the user's most original touch intent (two-dimensional screen coordinates) and converts it into standardized coordinates suitable for graphics computation through normalization processing. These standardized coordinates exist independently of the specific screen resolution, providing accurate and unified input data for the entire process. Step S432 converts the two-dimensional point into a directional three-dimensional spatial exploration ray. Using camera parameters (defining the camera's intrinsic properties, such as the frustum) and the real-time camera pose matrix provided by the SLAM system (defining the camera's position and orientation in three-dimensional space) as input, and then through back projection calculation, the standardized two-dimensional point provided in step S431 is endowed with three-dimensional depth information, transforming it into an infinitely extending ray originating from the camera's focal point and pointing in the direction pointed by the user, thus linking the two-dimensional screen of the mobile terminal with the real three-dimensional world. Step S433 anchors the user's touch intent in the real world by solving the intersection of the ray in step S432 with the real environment, and performs spatial geometric intersection calculation with the 3D mesh model of the environment pre-built or in real time by the SLAM system, so that the user's click operation on the mobile terminal screen corresponds to 3D coordinates in the real environment.

[0102] Through the optimized technical solutions provided in steps S431 to S433, users can interact with the augmented reality environment with high precision by clicking on the mobile terminal screen. For example, if a user clicks on a location on the real ground on the screen, the mobile terminal can immediately calculate the three-dimensional coordinates of that point, and then place a virtual piece of furniture, perform size measurements, or initiate spatial annotations at that location.

[0103] For the method steps disclosed in the above embodiments, the method steps are described as a series of actions for the purpose of simplicity. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0104] Any flowchart or other description of a process or method can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed and implemented not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, or in accordance with program structures such as loops, branches, etc., as will be readily understood by those skilled in the art when implementing the embodiments of this application.

[0105] like Figure 6 As shown, this application embodiment also provides an NFC haptic interactive display system that supports real-time query of building material processes, used to implement the NFC haptic interactive display method for supporting real-time query of building material processes provided in any specific embodiment of this application, including: The building material process audio and video loading module responds to the proximity event between the mobile terminal and the NFC tag of the building material process sample, reads the building material identification data stored in the NFC tag, obtains the corresponding building material process sample information from the server according to the building material identification data, the building material process sample information includes tactile waveform files and building material virtual model data, and loads the building material process audio and video content in the building material process sample information on the mobile terminal. Before playing audio and video content of building material process on the mobile terminal, the tactile waveform file node trigger generation module calls the audio clock interface of the mobile terminal and obtains the zero-buffered audio timestamp corresponding to the actual sound output time of the speaker. Based on the zero-buffered audio timestamp, a master clock reference is established, and the preset process node time value in the tactile waveform file is mapped to the master clock reference to generate a node trigger queue. The mobile terminal vibration control module, upon initiating playback, compares the current audio clock with the time value of the node trigger queue in real time through a registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback drive unit, causing the mobile terminal's vibration module to generate haptic vibration, and / or: The mobile terminal interactive output module captures the user's real environment using the mobile terminal's image acquisition device, calculates the embedding position and posture transformation matrix of the building material virtual model in the real space, superimposes the virtual model onto the real-time image stream based on the transformation matrix, and performs interactive output on the mobile terminal's display screen.

[0106] It is worth noting that although only some basic functional modules are disclosed in the embodiments of this application, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, the meaning expressed by the embodiments of this application is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules does not mean that the scope of protection of the claims of this application is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above device is described separately according to its functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0107] The implementation methods of the system described above are merely illustrative. For example, the various functional modules, units, or subsystems within the system may or may not be physically separate, or they may or may not be physical units; that is, they may be located in the same place or distributed across multiple different systems and their subsystems or modules. Those skilled in the art can select some or all of the functional modules, units, or subsystems to achieve the objectives of the embodiments of the present invention according to actual needs. Those skilled in the art can understand and implement the above-described situations without any creative effort.

[0108] like Figure 7 As shown, this application embodiment, in addition to providing an NFC tactile interactive display method and system that supports real-time query of building material processes, also provides corresponding electronic devices and storage media: Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The electronic device in this application includes: a processor, a non-volatile storage medium, internal memory, a network interface, and a system bus. The computer program can be stored in a non-volatile storage medium, which includes an operating system, a computer program, and a database. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, database, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM comes in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0109] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the relevant technical context and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0110] It should be noted that certain terms are used in the specification and claims of this application to refer to specific elements. Those skilled in the art will understand that different manufacturers or producers may use different terms to refer to the same element. The specification and claims of this application do not distinguish elements based on differences in terminology, but rather on differences in function.

[0111] In the description of the embodiments of this application, the reference to terms such as "an embodiment," "example," "specific example," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Furthermore, the technical solutions of the various implementation methods in this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the embodiments of this application.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing specific embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein, and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the specific embodiments of this application.

Claims

1. An NFC haptic interactive display method supporting real-time query of building material processes, characterized in that, include: In response to a proximity event between a mobile terminal and an NFC tag for a building material process sample, the system reads the building material identification data stored in the NFC tag, retrieves the corresponding building material process sample information from the server based on the building material identification data, and the building material process sample information includes a tactile waveform file, building material virtual model data, and building material process audio and video content. The system then loads the building material process audio and video content from the building material process sample information onto the mobile terminal. Before playing audio and video content about building material processes on the mobile terminal, the audio clock interface of the mobile terminal is called to obtain the zero-buffered audio timestamp corresponding to the actual sound output time of the speaker. A master clock reference is established based on the zero-buffered audio timestamp, and the preset process node time values ​​in the tactile waveform file are mapped to the master clock reference to generate a node trigger queue. After playback begins, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through the registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback drive unit, causing the vibration module of the mobile terminal to generate haptic vibration, and / or: The mobile terminal's image acquisition device captures the user's real-world environment, calculates the embedding position and pose transformation matrix of the virtual building material model in real space, overlays the virtual model onto a real-time image stream based on the transformation matrix, and interactively outputs it on the mobile terminal's display screen, further including: The pose transformation matrix and building material virtual model data are input into the graphics rendering engine. The spatial transformation of the model is realized through the vertex shader, the material lighting calculation is completed by the fragment shader, and the rendered virtual model is synthesized with the real-time environment image stream through alpha blending. The augmented reality image is then output on the mobile terminal display screen.

2. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 1, characterized in that, The step of loading the audio and video content of building material process in the building material process sample information on the mobile terminal further includes: the mobile terminal sending a query request to the server containing the building material identification data, wherein the query request encapsulates the device configuration information of the mobile terminal. The server receives the query request, performs a query operation in one or more building material information databases, and retrieves the associated benchmark building material process sample information based on the building material identification data; The server performs adaptive processing on the benchmark building material process sample information based on the received device configuration information. The adaptive processing specifically includes: generating a compatible tactile waveform file based on the tactile feedback capability of the mobile terminal, obtaining the device graphics rendering capability of the mobile terminal, selecting the corresponding building material virtual model data from the multi-level detail model library, and selecting a simplified model and using compressed textures for devices with low rendering capabilities. The server encapsulates the adaptively processed building material process sample information into a response data packet and transmits it back to the mobile terminal via the network.

3. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 2, characterized in that, The device configuration information includes: device model, operating system version, haptic feedback protocol version, graphics rendering capability level, and device haptic feedback capability. The server receives a request uploaded by a mobile terminal. The request includes device configuration information. The server adaptively processes the building material process sample information based on the device configuration information and encapsulates the processed virtual building material model data and an environmental 3D point cloud map into a response data packet. The mobile terminal receives and parses the response data packet, loads the environmental 3D point cloud map, identifies a physical plane as a reference in the loaded environmental 3D point cloud map, establishes a world coordinate system aligned with the physical plane, determines the virtual building material model in the world coordinate system, and completes the spatial registration of the virtual building material model. The mobile terminal acquires a video stream of real-time building material process sample information and solves the six-degree-of-freedom pose of the camera relative to the loaded 3D point cloud map of the environment through a visual tracking algorithm. Using six degrees of freedom pose and combined with camera intrinsics, the three-dimensional vertices of the virtual model that has completed spatial registration are used to calculate the two-dimensional coordinates of the vertices on the current image plane through perspective projection transformation. The virtual model is rendered in real time to generate a virtual image layer, and the virtual image layer is superimposed and synthesized with the real-time video stream to output the superimposed and synthesized augmented reality image.

4. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 1, characterized in that, Before playing the audio-visual content of building material processes on the mobile terminal, the process further includes: calling the audio clock interface of the mobile terminal to obtain a zero-buffered audio timestamp corresponding to the actual sound output time of the speaker; establishing a master clock reference based on the zero-buffered audio timestamp; mapping the preset process node time values ​​in the tactile waveform file to the master clock reference; and generating a node trigger queue. When starting the playback of audio and video content related to building material processes, the system submits the first frame audio data block to the audio hardware interface, obtains the expected presentation timestamp of the first frame audio data block returned by the audio system of the mobile terminal, and establishes a master clock reference synchronized with the audio hardware based on the expected presentation timestamp. The tactile waveform file in the building material process sample information is parsed, the process node time parameters in the tactile waveform file are extracted, and the node time values ​​are converted to the master clock reference coordinate system through a time mapping algorithm to generate a process node trigger queue based on time sequence arrangement. During playback, the playback progress is monitored in real time through an audio clock callback mechanism. When the time difference between the current playback time and the target time in the process node trigger queue is less than a set threshold, a trigger command is sent to the haptic drive unit to achieve synchronous audio playback and haptic feedback of building material process content.

5. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 4, characterized in that, The process further includes: parsing the tactile waveform file in the building material process sample information, extracting the process node time parameters from the tactile waveform file, converting the node time values ​​to the master clock reference coordinate system using a time mapping algorithm, and generating a process node trigger queue based on time sequence arrangement. Parse the metadata segment of the haptic waveform file, extract the process node time parameters in the metadata segment with the media timeline as the reference, and obtain the relative time offset of each process node; Based on the zero-buffered timestamp obtained from the audio system of the mobile terminal, a system-level master clock reference is established, and the relative time offset is converted into an absolute time coordinate based on the master clock using a time mapping function. By adding the relative time offset of each process node to the zero-buffer timestamp, the trigger time of all nodes in the absolute time coordinate system is calculated, and a sorted process node trigger queue is generated in chronological order.

6. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 1, characterized in that, After playback is initiated, the mobile terminal compares the current audio clock with the time value of the node trigger queue in real time through the registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback driving unit to cause the vibration module of the mobile terminal to generate haptic vibration, further including: After the media player on the mobile terminal starts, a high-precision clock callback function is registered with the audio system of the mobile terminal to establish a real-time monitoring channel based on the hardware audio clock. The callback function periodically obtains the sampled value of the current audio clock counter, reads the target time of the next node to be triggered from the node trigger queue, and calculates the time difference between the current clock value and the target time. When the time difference is detected to be less than the preset trigger threshold, a trigger command containing waveform parameters is sent to the haptic feedback drive unit, which then drives the vibration module to execute the corresponding haptic effect.

7. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 6, characterized in that, After the media player on the mobile terminal starts, registering a high-precision clock callback function with the audio system of the mobile terminal to establish a real-time monitoring channel based on the hardware audio clock further includes: After the media player completes initialization, it registers a high-precision clock callback function based on hardware interrupt driver by calling the underlying audio service interface of the operating system. Configure the execution parameters of the callback function, including setting the sampling interval, specifying the priority level, and allocating a dedicated interrupt handling thread; Each time the callback function is called, the corresponding system audio time is obtained by reading the current value of the audio hardware clock counter, and the system audio time is sent to the trigger determination module. The trigger determination module is used to determine whether the mobile terminal needs to be triggered to vibrate at the current moment.

8. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 1, characterized in that, The image acquisition device of the mobile terminal captures the user's real environment, calculates the embedding position and pose transformation matrix of the building material virtual model in real space, superimposes the virtual model onto the real-time image stream according to the transformation matrix, and performs interactive output on the mobile terminal's display screen, further including: The mobile terminal's image acquisition device is activated to capture the image stream of the real environment, the SLAM engine is initialized, and the six-degree-of-freedom pose of the mobile terminal in physical space is tracked in real time by fusing data from the camera and the inertial measurement unit. Based on the user's input position command, the two-dimensional screen coordinates are converted into three-dimensional spatial coordinates using a ray casting algorithm. The embedding position of the virtual model in the real environment is obtained through spatial geometric calculations, and the posture transformation matrix that transforms the model from the local coordinate system to the world coordinate system is calculated based on the embedding position.

9. The NFC haptic interactive display method for supporting real-time query of building material processes according to claim 1, characterized in that, The process of inputting the pose transformation matrix and building material virtual model data into a graphics rendering engine, realizing spatial transformation of the model through a vertex shader, performing material lighting calculations using a fragment shader, compositing the rendered virtual model with the real-time environment image stream through alpha blending, and outputting augmented reality images on a mobile terminal display screen further includes: By receiving touch input events from users on the mobile terminal touch screen, the two-dimensional coordinate data of the touch point in the screen coordinate system is obtained, and the two-dimensional coordinate data is normalized to the projection coordinate system. Based on the current camera parameters of the mobile terminal and the camera pose matrix provided by the SLAM system, the normalized screen coordinates are converted into three-dimensional spatial rays originating from the camera focus through back projection calculation. Using a spatial geometry intersection algorithm, the intersection points of the three-dimensional spatial ray and the three-dimensional mesh model of the environment constructed by the SLAM system are calculated, and the three-dimensional spatial coordinates of the first valid intersection point are output as the transformation result.

10. An NFC haptic interactive display system supporting real-time query of building material processes, used to implement the NFC haptic interactive display method supporting real-time query of building material processes as described in any one of claims 1 to 9, characterized in that, include: The building material process audio and video loading module responds to the proximity event between the mobile terminal and the NFC tag of the building material process sample, reads the building material identification data stored in the NFC tag, obtains the corresponding building material process sample information from the server based on the building material identification data, the building material process sample information includes tactile waveform files, building material virtual model data and building material process audio and video content, and loads the building material process audio and video content in the building material process sample information on the mobile terminal; Before playing audio and video content of building material process on the mobile terminal, the tactile waveform file node trigger generation module calls the audio clock interface of the mobile terminal and obtains the zero-buffered audio timestamp corresponding to the actual sound output time of the speaker. Based on the zero-buffered audio timestamp, a master clock reference is established, and the preset process node time value in the tactile waveform file is mapped to the master clock reference to generate a node trigger queue. The mobile terminal vibration control module, upon initiating playback, compares the current audio clock with the time value of the node trigger queue in real time through a registered audio clock callback function and obtains the difference. When the difference is less than a set threshold, a trigger signal is sent to the haptic feedback drive unit, causing the mobile terminal's vibration module to generate haptic vibration, and / or: A mobile terminal interactive output module, wherein the mobile terminal's image acquisition device captures the user's real-world environment, calculates the embedding position and pose transformation matrix of the virtual building material model in real space, superimposes the virtual model onto a real-time image stream based on the transformation matrix, and performs interactive output on the mobile terminal's display screen, further comprising: The pose transformation matrix and building material virtual model data are input into the graphics rendering engine. The spatial transformation of the model is realized through the vertex shader, the material lighting calculation is completed by the fragment shader, and the rendered virtual model is synthesized with the real-time environment image stream through alpha blending. The augmented reality image is then output on the mobile terminal display screen.

Images