An ar recognition and interaction method based on books and physical toys

Abstract

This invention discloses an AR recognition and interaction method based on books and physical toys. The method includes the following steps: offline feature registration; online recognition and pose estimation; bidirectional state synchronization; and offline and tracking compensation. This invention enables rapid, label-free recognition of book pages and physical toys, accurate six-DOF pose calculation, real-time linkage between physical entity operations and virtual character states, and tactile feedback from virtual events to physical toys. It ensures the system is fully usable offline, interference-resistant, and has stable tracking, significantly improving the naturalness, immersion, and practicality of the interaction, while also adapting to the offline operation requirements of mobile terminals.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to an AR recognition and interaction method based on books and physical toys. Background Technology

[0002] Augmented reality (AR) technology provides users with rich interactive experiences by overlaying computer-generated virtual information onto the real world, demonstrating enormous potential in education, entertainment, marketing, and other fields. Currently, most mainstream AR interaction methods rely on predefined image markers (such as QR codes or specific images) to trigger preset, single virtual content. Users scan these markers with their mobile devices to see the associated 3D models, animations, or videos on the screen.

[0003] However, existing technologies have significant shortcomings. First, the interaction is one-way and lagging. In existing technologies, physical objects, such as toys and books, are typically only used as "switches" to trigger virtual content. Once the virtual content is activated, its subsequent state changes, such as the character's actions and position, lack real-time correlation with the user's physical operations on the physical object, such as moving or rotating the toy. This results in a significant delay in the virtual character's response to physical operations, making it impossible to achieve bidirectional synchronization and real-time linkage between the physical entity and the digital twin, thus undermining the immersiveness and naturalness of the interaction. Second, there is an over-reliance on the network and weak offline capabilities. Many AR applications rely on cloud databases for image recognition and content downloading, requiring devices to be connected to the internet in real time. In environments with poor network signals or no network, such as remote areas or indoor dead corners, these applications will fail to start or frequently lag, severely limiting the use cases and reliability of AR products. Third, there is a lack of robustness in tag-based recognition. Reliance on QR codes or specific images limits the range of identifiable objects, and recognition is prone to failure if the marker is obscured, damaged, or in an environment with strong changes in lighting, resulting in lost tracking, disappearance or misalignment of virtual content, and a significant reduction in user experience.

[0004] To address the aforementioned problems, this invention proposes a lightweight, markerless AR recognition and interaction method based on books and physical toys, aiming to solve the technical problems of difficulty in bidirectional synchronization between physical toys and virtual content, strong network dependence, and easy loss of tracking. Summary of the Invention

[0005] This invention provides an AR recognition and interaction system based on books and physical toys, the system comprising:

[0006] The offline feature registration module is used to perform offline scanning on at least one entity object, extract its local image features, calculate the three-dimensional coordinates of each local feature in the entity object's own coordinate system, generate a registration file containing feature point sets and their corresponding three-dimensional coordinate information, and build an offline feature library.

[0007] The online recognition and pose estimation module is used to acquire video frames containing the entity object in real time, extract local image features of the current frame, and match them with the feature point set in the offline feature library to confirm the identity of the entity object; based on the successfully matched 2D-2D or 2D-3D feature correspondence, the six degrees of freedom pose information of the entity object relative to the acquisition device is calculated.

[0008] The bidirectional state synchronization module is used to drive the state of the virtual twin corresponding to the physical object to be updated in real time based on the six degrees of freedom attitude information, so as to realize the forward synchronization from the physical entity to the virtual twin; at the same time, it monitors the events that occur in the virtual scene and generates tactile control signals according to the events, and sends them to the tactile feedback device embedded in the physical object, so as to realize the reverse synchronization from the virtual twin to the physical object.

[0009] The offline and tracking compensation module is used to cache key frames and feature point maps of the current scene locally after the first successful identification. When the network is disconnected or tracking is briefly lost, the module uses a motion model to predict the current pose of the entity object and continuously compares the global descriptor of the current frame with the locally cached scene descriptor. After successful relocalization, the correct display of the virtual content is restored.

[0010] Furthermore, the offline feature registration module is further used to perform dimensionality reduction processing on the extracted local features of the image to generate a compressed descriptor. Specifically, this includes: using a scale-invariant feature transformation algorithm or an improved algorithm to detect key points and construct a high-dimensional feature vector, and then using principal component analysis to reduce the high-dimensional feature vector to a preset dimension to obtain the compressed descriptor.

[0011] Furthermore, the online recognition and pose estimation module calculates the six-degree-of-freedom pose information of the entity object relative to the acquisition device by: minimizing the reprojection error, solving for the rotation matrix and translation vector from the entity object's own coordinate system to the camera coordinate system, wherein the reprojection error is the deviation between the pixel coordinates of the matching point on the real-time image and the coordinates of the corresponding three-dimensional point projected onto the image plane according to the rotation matrix and translation vector.

[0012] Furthermore, the bidirectional state synchronization module realizes forward synchronization from the physical entity to the virtual twin, specifically including: performing Kalman filtering on the continuously calculated six-degree-of-freedom attitude information to filter out noise and predict the attitude at the next moment, and using the filtered attitude parameters to drive the state update of the virtual twin.

[0013] Furthermore, the Kalman filtering process includes a prediction stage and an update stage. The prediction stage calculates the state prior estimate at the current time based on the state posterior estimate and the state transition matrix at the previous time. The update stage corrects the state prior estimate by combining the actual observations at the current time to obtain the state posterior estimate at the current time.

[0014] Furthermore, during the reverse synchronization process of the bidirectional state synchronization module, the tactile control signal is a modulated sine wave signal, the amplitude and frequency of which are quantized and determined according to the type and intensity of events occurring in the virtual scene.

[0015] Furthermore, the offline tracking compensation module uses a motion model to predict the current pose of the entity object. Specifically, it multiplies the pose estimate from the last successful tracking by a predicted rotation increment matrix to obtain the predicted pose at the current moment.

[0016] This invention also provides an AR recognition and interaction method based on books and physical toys. This method is used in the aforementioned system and includes the following steps:

[0017] S1. Perform offline scanning on at least one entity object, extract its local image features, calculate the three-dimensional coordinates of each local feature in the entity object's own coordinate system, generate a registration file containing feature point sets and their corresponding three-dimensional coordinate information, and construct an offline feature library.

[0018] S2. Real-time acquisition of video frames containing the entity object, extraction of local image features of the current frame, and matching of these features with the feature point set in the offline feature library to confirm the identity of the entity object; based on the successfully matched 2D-2D or 2D-3D feature correspondence, calculation of the six degrees of freedom pose information of the entity object relative to the acquisition device;

[0019] S3. Based on the six degrees of freedom posture information, drive the state of the virtual twin corresponding to the physical object to be updated synchronously in real time, so as to realize the forward synchronization from the physical entity to the virtual twin; at the same time, monitor the events that occur in the virtual scene, and generate tactile control signals according to the events, and send them to the tactile feedback device embedded in the physical object, so as to realize the reverse synchronization from the virtual twin to the physical object.

[0020] S4. After successful initial recognition, keyframes and feature point maps of the current scene are cached locally. When the network is disconnected or tracking is briefly lost, the current pose of the entity is predicted using a motion model, and the global descriptor of the current frame is continuously compared with the scene descriptor cached locally. After successful relocation, the correct display of the virtual content is restored.

[0021] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, which is executed by a processor to perform the above-described AR recognition and interaction method based on books and physical toys.

[0022] The present invention also provides a computer program product, which includes a computer program executed by a processor for performing the above-described AR recognition and interaction method based on books and physical toys.

[0023] This invention constructs a lightweight feature library through offline multimodal feature registration, enabling rapid, label-free recognition of book pages and physical toys, as well as accurate six-degree-of-freedom pose calculation. Utilizing a Kalman filter-optimized bidirectional state synchronization mechanism, it achieves real-time linkage between physical entity operations and virtual character states, and tactile feedback from virtual events to physical toys. Combined with local map persistence and tracking loss reconnection algorithms, it ensures the system is fully usable offline, interference-resistant, and has stable tracking. This effectively solves the technical problems of existing AR products, such as reliance on markers, one-way interaction, network dependence, and easy tracking loss, significantly improving the naturalness, immersion, and practicality of the interaction, while also adapting to the offline operation requirements of mobile terminals. Attached Figure Description

[0024] Figure 1 This is a flowchart of an AR recognition and interaction method based on books and physical toys according to the present invention. Detailed Implementation

[0025] The present invention will now be further described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention. It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention.

[0026] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Embodiment 1 of the present invention relates to an AR recognition and interaction system based on books and physical toys, comprising:

[0028] The offline feature registration module is used to perform offline scanning on at least one entity object, extract its local image features, calculate the three-dimensional coordinates of each local feature in the entity object's own coordinate system, generate a registration file containing feature point sets and their corresponding three-dimensional coordinate information, and build an offline feature library.

[0029] The online recognition and pose estimation module is used to acquire video frames containing the entity object in real time, extract local image features of the current frame, and match them with the feature point set in the offline feature library to confirm the identity of the entity object; based on the successfully matched 2D-2D or 2D-3D feature correspondence, the six-degree-of-freedom pose information of the entity object relative to the acquisition device is calculated.

[0030] The bidirectional state synchronization module is used to drive the state of the virtual twin corresponding to the physical object to be updated in real time based on the six degrees of freedom attitude information, so as to achieve forward synchronization from the physical entity to the virtual twin; at the same time, it monitors the events that occur in the virtual scene and generates tactile control signals according to the events, and sends them to the tactile feedback device embedded in the physical object, so as to achieve reverse synchronization from the virtual twin to the physical object.

[0031] The offline and tracking compensation module is used to cache key frames and feature point maps of the current scene locally after the first successful identification. When the network is disconnected or tracking is briefly lost, the module uses a motion model to predict the current pose of the entity object and continuously compares the global descriptor of the current frame with the locally cached scene descriptor. After successful relocalization, the correct display of the virtual content is restored.

[0032] The physical objects include book pages and / or three-dimensional toys.

[0033] The offline feature registration module is further used to perform dimensionality reduction processing on the extracted local features of the image to generate a compressed descriptor. Specifically, it includes: using a scale-invariant feature transformation algorithm or an improved algorithm to detect key points and construct a high-dimensional feature vector, and then using principal component analysis to reduce the high-dimensional feature vector to a preset dimension to obtain the compressed descriptor.

[0034] The online recognition and attitude estimation module calculates the six-degree-of-freedom attitude information of the entity object relative to the acquisition device by minimizing the reprojection error, and solves the rotation matrix and translation vector from the entity object's own coordinate system to the camera coordinate system. The reprojection error is the deviation between the pixel coordinates of the matching point on the real-time image and the coordinates of the corresponding three-dimensional point projected onto the image plane according to the rotation matrix and translation vector.

[0035] The bidirectional state synchronization module realizes forward synchronization from the physical entity to the virtual twin, specifically including: performing Kalman filtering on the continuously calculated six-degree-of-freedom attitude information to filter out noise and predict the attitude at the next moment, and using the filtered attitude parameters to drive the state update of the virtual twin.

[0036] The Kalman filtering process includes a prediction stage and an update stage. In the prediction stage, the state prior estimate at the current time is calculated based on the state posterior estimate and the state transition matrix at the previous time. In the update stage, the state prior estimate is corrected by combining the actual observations at the current time to obtain the state posterior estimate at the current time.

[0037] During the reverse synchronization process of the bidirectional state synchronization module, the tactile control signal is a modulated sine wave signal, and its amplitude and frequency are quantized and determined according to the type and intensity of the events occurring in the virtual scene.

[0038] The offline tracking compensation module uses a motion model to predict the current pose of the entity object. Specifically, it multiplies the pose estimate from the last successful tracking by a predicted rotation increment matrix to obtain the predicted pose at the current moment.

[0039] Embodiment 2 of the present invention relates to an AR recognition and interaction method based on books and physical toys, as shown in the appendix. Figure 1 As shown, the method includes the following steps:

[0040] S1. Perform offline scanning on at least one entity object, extract its local image features, calculate the three-dimensional coordinates of each local feature in the entity object's own coordinate system, generate a registration file containing feature point sets and their corresponding three-dimensional coordinate information, and construct an offline feature library.

[0041] S2. Real-time acquisition of video frames containing the entity object, extraction of local image features of the current frame, and matching of these features with the feature point set in the offline feature library to confirm the identity of the entity object; based on the successfully matched 2D-2D or 2D-3D feature correspondence, calculation of the six degrees of freedom pose information of the entity object relative to the acquisition device.

[0042] S3. Based on the six degrees of freedom attitude information, the state of the virtual twin corresponding to the physical object is updated synchronously in real time to achieve forward synchronization from the physical entity to the virtual twin; at the same time, events occurring in the virtual scene are monitored, and tactile control signals are generated according to the events and sent to the tactile feedback device embedded in the physical object to achieve reverse synchronization from the virtual twin to the physical object.

[0043] S4. After successful initial recognition, keyframes and feature point maps of the current scene are cached locally. When the network is disconnected or tracking is briefly lost, the current pose of the entity is predicted using a motion model, and the global descriptor of the current frame is continuously compared with the scene descriptor cached locally. After successful relocation, the correct display of the virtual content is restored.

[0044] For step S1, this step can be completed before printing and / or production of physical objects, such as book pages and / or three-dimensional toys. For each physical object, a unique identifier is constructed through offline scanning, and an offline feature library is built by entering its identity information. Specifically, this includes: processing the template image using an improved Scale Invariant Feature Transform (SIFT) descriptor to extract local image features; and combining Gaussian difference pyramids for extreme point detection when detecting key points to ensure scale invariance. For each key point, a 128-dimensional feature vector d is constructed. Then, to adapt to offline devices and reduce storage space, principal component analysis is used to reduce the dimensionality of the feature vector d to obtain a compressed scan vector.

[0045]

[0046] in, This is the final compressed descriptor; Principal component analysis is used as a dimensionality reduction function to reduce high-dimensional eigenvectors to kk dimensions. The target dimension after dimensionality reduction; An improved SIFT feature extraction function that integrates the difference-of-Gaussian pyramids is used to generate scale-invariant feature vectors. The template image is the input. The scaling parameter of the difference-of-Gaussian pyramid; This represents the number of levels in the Gaussian difference pyramid.

[0047] For three-dimensional objects such as toys, the known camera intrinsic matrix and multi-view geometry are used to calculate the three-dimensional spatial coordinates of each feature point through triangulation. This ultimately generates a registration file M containing N elements.

[0048]

[0049] in, This is the feature point set of the k-th registered object; This is a compressed descriptor for the i-th feature point; Let i be the three-dimensional coordinates of the i-th feature point in the physical entity's own coordinate system. The total number of feature points for the k-th object.

[0050]

[0051] Where M is the set of registration files for all registered objects; The number of registered objects; Let be the feature point set of the k-th registered object.

[0052] For step S2, when the user scans with the device, the system calculates the identity of the current physical entity and its pose relative to the camera in real time, specifically including:

[0053] S21. Feature matching and identity verification.

[0054] Extracting feature points from real-time video frames The algorithm uses a fast nearest neighbor search (FRASS) to find the nearest and second nearest neighbors in the registry M, and then filters out mismatches using a ratio test.

[0055]

[0056] in, This is a compressed descriptor for the j-th feature point extracted from the current real-time video frame; To be compatible with the pre-built registry library M The compressed descriptor corresponding to the feature point with the closest Euclidean distance; To be compatible with the pre-built registry library M The compressed descriptor corresponding to the feature point with the second closest Euclidean distance. Let be the norm of the vector; This is a preset ratio threshold; This is the ratio of the calculated nearest distance to the second nearest distance; the smaller this value, the higher the accuracy of the current match. If... If the value is less than τ, then the match is considered correct, and the corresponding point pair is accepted.

[0057] S22. Six-DOF attitude calculation.

[0058] By utilizing the 2D-3D correspondence obtained through matching, the rotation matrix R and translation vector t of the camera coordinate system relative to the physical entity coordinate system are solved. This solution is achieved by minimizing the reprojection error. This yields the complete six-degree-of-freedom pose, addressing the question of "where the physical entity is and how it is positioned," which is fundamental to achieving virtual-real synchronization.

[0059]

[0060] in, This represents the reprojection error value. The rotation matrix describes the rotation transformation from the physical entity's own coordinate system to the camera's coordinate system; It is a translation vector that describes the translation from the origin of the physical entity's own coordinate system to the origin of the camera's coordinate system. Let be the pixel coordinates of the j-th matching point in the real-time image; This is the camera projection function, which projects 3D points in the camera coordinate system onto the image plane; This is the camera intrinsic parameter matrix; To and The coordinates of the three-dimensional point on the physical entity model corresponding to the i-th successfully matched point.

[0061] Step S3 is used to achieve state consistency between the physical entity and the digital twin. Specifically, it includes:

[0062] S31. Define the system state vector , representing position, velocity, attitude quaternions and angular velocity, respectively.

[0063] Achieving forward synchronization from physical entities to virtual twins specifically includes:

[0064] S32. Data acquisition: Real-time calculation of attitude changes through visual tracking.

[0065] S33. Kalman Filter Prediction: Considering noise from handshakes or rapid movement, a linear Kalman filter is used to predict the attitude at the next moment. The filter equation is established as follows:

[0066]

[0067]

[0068] in, Let be the posterior state estimate vector at time k-1; Let be the prior state estimate vector at time k; This is the state transition matrix; To control the input matrix; To control the input vector; Let k be the state error covariance matrix at time k-1; Let k be the state error covariance matrix at time k; Let be the process noise covariance matrix.

[0069] S34. Establish the update equation:

[0070]

[0071]

[0072]

[0073] in, The observation matrix maps the state vector to the observation space; To observe the noise covariance matrix; Let k be the state error covariance matrix at time k; The Kalman gain matrix; The actual observed value at time k; Let be the posterior state estimation vector at time k; Let be the prior state estimate vector at time k; This is the posterior estimate of the state error covariance matrix at time k; It is an identity matrix.

[0074] S35. Perform virtual character driving, map the filtered posture parameters R to the skeletal nodes of the 3D character, and realize the visual effect of the physical entity and the virtual twin moving simultaneously.

[0075] The reverse synchronization from virtual twin to physical entity specifically includes:

[0076] S36. Introduce a tactile encoding and decoding mechanism to enable simultaneous movement of the virtual twin and the physical entity, including event quantization. Based on collisions or actions occurring in the virtual scene, the system generates an intensity signal. and frequency Signal modulation sends the signal to a miniature linear resonant motor embedded in a physical entity, controlling the signal. for:

[0077]

[0078] in, This is the control signal sent to the micro linear resonant motor at time t; The amplitude; t is the vibration frequency; t is the time variable; is the envelope function, used to modulate the amplitude of the carrier wave as a function of time; This is the phase parameter.

[0079] For step S4, when disconnection or temporary loss of tracking occurs, the current pose of the entity object is predicted using a motion model, specifically including:

[0080] When the network disconnects or tracking is briefly lost, the rotation matrix of the object at the next moment is predicted using a motion model. Simultaneously, the global descriptor of the current frame is continuously extracted and compared with the locally cached scene descriptor. Once relocalization is successful, the virtual content is immediately restored to the correct position, including calculations such as:

[0081]

[0082] in, Let t be the rotation matrix of the physical entity at time t, which is the attitude estimate at the time of the last successful tracking. This is the predicted rotation increment matrix; This is the rotation matrix of the physical entity at the predicted next time step t+1.

[0083] Embodiment 3 of the present invention relates to a computer-readable storage medium for storing a computer program, which is executed by a processor to perform the method of Embodiment 1.

[0084] Embodiment 4 of the present invention relates to a computer program product, the computer program product comprising a computer program executed by a processor for performing the method of Embodiment 2.

[0085] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An AR recognition and interaction system based on books and physical toys, characterized in that, The system includes: The offline feature registration module is used to perform offline scanning on at least one entity object, extract its local image features, calculate the three-dimensional coordinates of each local feature in the entity object's own coordinate system, generate a registration file containing feature point sets and their corresponding three-dimensional coordinate information, and build an offline feature library. The online recognition and pose estimation module is used to acquire video frames containing the entity object in real time, extract local image features of the current frame, and match them with the feature point set in the offline feature library to confirm the identity of the entity object; based on the successfully matched 2D-2D or 2D-3D feature correspondence, the six degrees of freedom pose information of the entity object relative to the acquisition device is calculated. The bidirectional state synchronization module is used to drive the state of the virtual twin corresponding to the physical object to be updated in real time based on the six degrees of freedom attitude information, so as to realize the forward synchronization from the physical entity to the virtual twin; at the same time, it monitors the events that occur in the virtual scene and generates tactile control signals according to the events, and sends them to the tactile feedback device embedded in the physical object, so as to realize the reverse synchronization from the virtual twin to the physical object. The offline and tracking compensation module is used to cache key frames and feature point maps of the current scene locally after the first successful identification. When the network is disconnected or tracking is briefly lost, the module uses a motion model to predict the current pose of the entity object and continuously compares the global descriptor of the current frame with the locally cached scene descriptor. After successful relocalization, the correct display of the virtual content is restored.

2. The AR recognition and interaction system based on books and physical toys according to claim 1, characterized in that, The offline feature registration module is further used to perform dimensionality reduction processing on the extracted local features of the image to generate a compressed descriptor. Specifically, it includes: using a scale-invariant feature transformation algorithm or an improved algorithm to detect key points and construct a high-dimensional feature vector, and then using principal component analysis to reduce the high-dimensional feature vector to a preset dimension to obtain the compressed descriptor.

3. The AR recognition and interaction system based on books and physical toys according to claim 1, characterized in that, The online recognition and attitude estimation module calculates the six-degree-of-freedom attitude information of the entity object relative to the acquisition device by minimizing the reprojection error, and solves the rotation matrix and translation vector from the entity object's own coordinate system to the camera coordinate system. The reprojection error is the deviation between the pixel coordinates of the matching point on the real-time image and the coordinates of the corresponding three-dimensional point projected onto the image plane according to the rotation matrix and translation vector.

4. The AR recognition and interaction system based on books and physical toys according to claim 1, characterized in that, The bidirectional state synchronization module realizes forward synchronization from the physical entity to the virtual twin, specifically including: performing Kalman filtering on the continuously calculated six-degree-of-freedom attitude information to filter out noise and predict the attitude at the next moment, and using the filtered attitude parameters to drive the state update of the virtual twin.

5. The AR recognition and interaction system based on books and physical toys according to claim 1, characterized in that, The Kalman filtering process includes a prediction stage and an update stage. In the prediction stage, the state prior estimate at the current time is calculated based on the state posterior estimate and the state transition matrix at the previous time. In the update stage, the state prior estimate is corrected by combining the actual observations at the current time to obtain the state posterior estimate at the current time.

6. The AR recognition and interaction system based on books and physical toys according to claim 1, characterized in that, During the reverse synchronization process of the bidirectional state synchronization module, the tactile control signal is a modulated sine wave signal, and its amplitude and frequency are quantized and determined according to the type and intensity of the events occurring in the virtual scene.

7. The AR recognition and interaction system based on books and physical toys according to claim 1, characterized in that, The offline tracking compensation module uses a motion model to predict the current pose of the entity object. Specifically, it multiplies the pose estimate from the last successful tracking by a predicted rotation increment matrix to obtain the predicted pose at the current moment.

8. An AR recognition and interaction method based on books and physical toys, the method being used in the system described in any one of claims 1-7, characterized in that, The method includes the following steps: S1. Perform offline scanning on at least one entity object, extract its local image features, calculate the three-dimensional coordinates of each local feature in the entity object's own coordinate system, generate a registration file containing feature point sets and their corresponding three-dimensional coordinate information, and construct an offline feature library. S2. Real-time acquisition of video frames containing the entity object, extraction of local image features of the current frame, and matching of these features with the feature point set in the offline feature library to confirm the identity of the entity object; based on the successfully matched 2D-2D or 2D-3D feature correspondence, calculation of the six degrees of freedom pose information of the entity object relative to the acquisition device; S3. Based on the six degrees of freedom posture information, drive the state of the virtual twin corresponding to the physical object to be updated synchronously in real time, so as to realize the forward synchronization from the physical entity to the virtual twin; at the same time, monitor the events that occur in the virtual scene, and generate tactile control signals according to the events, and send them to the tactile feedback device embedded in the physical object, so as to realize the reverse synchronization from the virtual twin to the physical object. S4. After successful initial recognition, keyframes and feature point maps of the current scene are cached locally. When the network is disconnected or tracking is briefly lost, the current pose of the entity is predicted using a motion model, and the global descriptor of the current frame is continuously compared with the scene descriptor cached locally. After successful relocation, the correct display of the virtual content is restored.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which is executed by a processor to perform an AR recognition and interaction method based on books and physical toys as described in claim 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that is executed by a processor to perform an AR recognition and interaction method based on books and physical toys as described in claim 8.

Images