AR (Augmented Reality) interaction method, system, equipment and medium

Abstract

The invention provides an AR interaction method, system, device and medium, and belongs to the technical field of augmented reality and smart museums, and the method comprises the following steps: initializing an AR scene, extracting a plurality of key points of a to-be-interacted object, and establishing a three-dimensional tracking coordinate of the to-be-interacted object according to the plurality of key points; constructing a motion state transition matrix representing different states of the to-be-interacted object according to the three-dimensional tracking coordinates of the to-be-interacted object; on the basis of the motion state transition matrix, a motion track prediction equation of the to-be-interacted object is obtained in combination with a Kalman filter; acquiring a velocity vector and an acceleration vector of butterfly motion, and constructing a butterfly motion equation according to the velocity vector and the acceleration vector; and performing AR interaction between the to-be-interacted object and the butterfly according to the butterfly motion equation and the motion trail prediction equation of the to-be-interacted object. According to the invention, virtual and real fused interaction between the object to be interacted and the butterfly can be realized, a more visual and vivid display mode is provided, and the interaction experience of a user is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical fields of augmented reality and intelligent museums, and specifically relates to an AR interaction method, system, device, and medium. Background Art

[0002] Augmented reality (AR) technology is a technology that superimposes virtual information onto the real world in real time through computer technology, aiming to enhance users' perception and interaction experience of the real environment. With the improvement of the performance of mobile devices and the development of sensor technology, AR technology has been widely applied in many fields such as education, entertainment, and cultural display. Its core advantage lies in being able to achieve the combination of virtual and real, real-time interaction, and provide an immersive user experience, bringing a brand-new visual and interaction feeling to users.

[0003] In recent years, with the advancement of the construction of intelligent museums, the museum field has gradually introduced AR technology to enhance the interactivity and interest of exhibitions. Intelligent museums achieve the upgrading of cultural relic display, education dissemination, and audience service through digital and information means. Among them, AR technology, as an important display means, can present information such as the three-dimensional model and historical background of cultural relics to the audience in a virtual way, enabling the audience to obtain richer knowledge and experience during the visit. For example, some museums use AR technology to allow the audience to scan cultural relics through mobile phones or tablet devices, and then they can see the virtual restoration of the cultural relics or the reproduction of relevant historical scenes.

[0004] However, the application of AR technology in museums still has certain limitations at present. Most AR displays only stay at the static virtual presentation of cultural relics, lacking in-depth interaction with the audience, resulting in insufficient immersion of the audience and being difficult to meet the audience's needs for interactivity and interest. Summary of the Invention

[0005] In order to overcome the deficiencies of the above-mentioned existing technologies, the present invention provides an AR interaction method, including the following steps:

[0006] Initialize the AR scene, extract multiple key points of the object to be interacted with, and establish a three-dimensional tracking coordinate of the object to be interacted with according to the multiple key points;

[0007] Construct a motion state transition matrix representing different states of the object to be interacted with according to the three-dimensional tracking coordinate of the object to be interacted with;

[0008] Based on the motion state transition matrix, and in combination with a Kalman filter, obtain a motion trajectory prediction equation of the object to be interacted with;

[0009] Obtain the velocity vector and acceleration vector of the butterfly's movement, and construct a butterfly movement equation according to the velocity vector and acceleration vector;

[0010] Obtain the motion state vector of the object to be interacted with and the position vector of the butterfly, input the motion state vector of the object to be interacted with and the position vector of the butterfly into the motion trajectory prediction equation of the object to be interacted with and the butterfly motion equation respectively, and perform AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly motion equation.

[0011] The object to be interacted with includes gestures, faces, and full-body postures. Multiple key points of gestures are extracted by a 21-point hand key point model; multiple key points of faces are extracted using a 68-point feature point model; multiple key points of full-body postures are extracted based on a 17-point skeleton model of the COCO standard.

[0012] The specific form of the butterfly motion equation is as follows:

[0013] P(t) = P0 + vt + (1 / 2)at²;

[0014] Where v is the butterfly velocity vector, a is the butterfly acceleration vector. P0 is the initial position vector of the butterfly, t is the butterfly motion time parameter, and P(t) is the current position vector of the butterfly.

[0015] The specific form of the motion trajectory prediction equation of the object to be interacted with is as follows:

[0016] X(t|t - 1) = AX(t - 1) + BU(t) + w(t);

[0017] Where A is the motion state transition matrix, B is the motion control matrix, w(t) is the motion process noise, X is the motion state vector, t represents the time parameter, X(t|t - 1) is the motion prediction state vector, X(t - 1) is the motion history state vector, and U(t) is the motion control input vector.

[0018] After performing AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly motion equation, it further includes performing barrage interaction using a time and space scheduling algorithm, which is specifically implemented through a barrage motion equation. The specific form of the barrage motion equation is as follows:

[0019] P(t)' = P0' + v't';

[0020] Where P(t)' is the current position vector of the barrage, P0' is the initial position vector of the barrage, v' is the barrage velocity vector, and t' is the barrage motion time.

[0021] The present invention also provides an AR interaction system, including:

[0022] A key point extraction module, used to initialize the AR scene, extract multiple key points of the object to be interacted with, and establish a three-dimensional tracking coordinate of the object to be interacted with according to the multiple key points;

[0023] A motion state transition matrix acquisition module, configured to construct a motion state transition matrix representing different states of an object to be interacted based on the three-dimensional tracking coordinates of the object to be interacted;

[0024] A motion trajectory prediction equation acquisition module, configured to obtain a motion trajectory prediction equation of the object to be interacted based on the motion state transition matrix and in combination with a Kalman filter;

[0025] A butterfly motion equation construction module, configured to obtain a velocity vector and an acceleration vector of butterfly motion, and construct a butterfly motion equation according to the velocity vector and the acceleration vector;

[0026] An interaction module, configured to obtain a motion state vector of the object to be interacted and a position vector of the butterfly, input the motion state vector of the object to be interacted and the position vector of the butterfly into the motion trajectory prediction equation of the object to be interacted and the butterfly motion equation respectively, and perform AR interaction between the object to be interacted and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly motion equation.

[0027] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the AR interaction method.

[0028] The present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the AR interaction method.

[0029] The AR interaction method provided by the present invention has the following beneficial effects:

[0030] In the AR scenario of the present invention, multiple key points of the object to be interacted are extracted, and three-dimensional tracking coordinates of the object to be interacted are established according to the multiple key points, so as to represent a motion state transition matrix between different states of the object to be interacted; according to the motion state transition matrix and the Kalman filter, a motion trajectory prediction equation of the object to be interacted can be obtained, and the actions and positions of the object to be interacted can be tracked through the motion trajectory prediction equation; a butterfly motion equation is constructed according to the velocity vector and the acceleration vector; a butterfly motion equation is constructed according to the velocity vector and the acceleration vector of the butterfly motion, and AR interaction between the object to be interacted and the butterfly is performed according to the butterfly motion equation and the motion trajectory prediction equation of the object to be interacted. This process realizes the virtual-real blend interaction between the object to be interacted and the butterfly, provides a more intuitive and vivid display method, enables the user (the issuer of the object to be interacted instruction) to be in the dreamy world of the butterfly, and enhances the interaction experience between the user and the virtual butterfly. Description of the Drawings

[0031] To more clearly illustrate the embodiments of the present invention and their design solutions, the accompanying drawings required for this embodiment will be briefly introduced below. The drawings in the following description are only partial embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 is the flowchart of the present invention;

[0033] Figure 2 is the overall system architecture diagram of the present invention;

[0034] Figure 3 is the three-dimensional model diagram of part of a butterfly;

[0035] Figure 4 is the operation flowchart of AR gesture special effects;

[0036] Figure 5 is the effect diagram of AR gesture special effects;

[0037] Figure 6 is the operation flowchart of AR face special effects;

[0038] Figure 7 is the effect diagram of AR face special effects;

[0039] Figure 8 is the operation flowchart of AR full-body special effects;

[0040] Figure 9 is the effect diagram of AR full-body special effects;

[0041] Figure 10 is the operation flowchart of AR image special effects;

[0042] Figure 11 is the effect diagram of AR image special effects;

[0043] Figure 12 is the operation flowchart of the user message interaction function;

[0044] Figure 13 is the effect diagram of the user interaction function. Specific embodiments

[0045] In order to enable those skilled in the art to better understand the technical solutions of the present invention and be able to implement them, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0046] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "axial", "radial", "circumferential", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the technical solution of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention.

[0047] In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In the description of the present invention, unless otherwise stated, the meaning of "a plurality of" is two or more, which will not be elaborated here.

[0048] Embodiment

[0049] The present invention provides an AR interaction method, specifically as Figure 1 shown. It initializes the AR scene, extracts multiple key points of the object to be interacted with, and establishes a three-dimensional tracking coordinate of the object to be interacted with according to the multiple key points; constructs a motion state transition matrix representing different states of the object to be interacted with according to the three-dimensional tracking coordinate of the object to be interacted with; based on the motion state transition matrix and in combination with a Kalman filter, obtains a motion trajectory prediction equation of the object to be interacted with; obtains a velocity vector and an acceleration vector of the butterfly movement, and constructs a butterfly movement equation according to the velocity vector and the acceleration vector; obtains a motion state vector of the object to be interacted with and a position vector of the butterfly, inputs the motion state vector of the object to be interacted with and the position vector of the butterfly into the motion trajectory prediction equation of the object to be interacted with and the butterfly movement equation respectively, and performs AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly movement equation. To make it easier to understand the technical solution of the present invention, this embodiment will be described in detail through various interaction methods, specifically as follows (see Figure 2 ):

[0050] Construction of the data management module: It includes the Butterfly Model Database (Butterfly Digital Resource Library and Butterfly Digital Resource Library) and the User Information Database. Butterfly Model Database: Create and maintain a database containing various virtual butterfly 3D models, which is used to store and provide butterfly models in the AR experience (see Figure 3 ). User Information Database: Store the personal information and interaction history of users to provide personalized experiences for users.

[0051] Initialize the augmented reality environment: Through the camera and sensors of the user's mobile device (such as a mobile phone or tablet), obtain the real scene of the museum, and initialize the augmented reality environment based on the XR-Frame framework.

[0052] Provide multiple AR special effect options: Provide multiple special effect options such as AR gesture special effects, AR face special effects, AR full-body special effects, and AR image special effects in the user interface. Users can select the corresponding special effect modes according to their preferences.

[0053] Implement AR gesture special effect interaction: Utilize the gesture detection function to recognize the user's gesture actions (such as waving, clicking, etc.), and guide the virtual butterfly to fly or stay on the user's palm according to the gesture actions, enhancing the user's immersive interactive experience.

[0054] Implement AR face special effect interaction: Through face recognition technology, dynamically add butterfly wings, flower wreaths, or other decorative elements to the user's face, and implement the special effect of butterflies flying around the face, adding fun to the user's selfies and short videos.

[0055] Implement AR full-body special effect interaction: Through full-body recognition technology, real-time track the user's actions and positions, enabling the user to see a scene where they are surrounded by butterflies, as if being in a butterfly garden. At the same time, the virtual butterflies can make corresponding interactive responses according to the user's actions.

[0056] Implement AR image special effect interaction: Use augmented reality technology to superimpose vivid butterflies on the real world, allowing users to see the interaction between the butterflies and the surrounding environment on the screen, such as butterflies staying on flowers or flying in the air, creating a dreamy visual effect.

[0057] Provide user interaction functions: Set special effect buttons, comment buttons, bullet screen buttons, photo-taking buttons, and recording buttons in the user interface. Users can select different butterfly special effects by clicking the special effect buttons, express their feelings through the comment buttons, display comments in the form of bullet screens through the bullet screen buttons, and save their experiences through the photo-taking and recording buttons.

[0058] Data Storage and Feedback: The user's interaction data and experience feedback are stored and managed through a cloud server. At the same time, a user experience questionnaire is provided to collect the opinions and suggestions of users for system optimization and improvement.

[0059] The AR gesture special effect interaction includes the following steps:

[0060] First, initialize the AR scene through the xr-scene component and establish a three-dimensional tracking coordinate system {X, Y, Z}. The scene rendering uses a combined transformation of the projection matrix P and the view matrix V: M = P × V, where the near clipping plane parameter near = 0.01 to ensure millimeter-level rendering accuracy. The rendering pipeline is implemented through WebGL, and the frame rendering time Tf satisfies: Tf ≤ 1 / 60 second.

[0061] Second, the resource management module adopts an asynchronous loading strategy, and the relationship between the loading time T and the resource size S satisfies: T = S / B + L, where B is the bandwidth and L is the network latency. The model index is implemented through the dynamic resource identifier f(x) = "butterfly" + x, where x ∈ [1, 6] represents the special effect type. The resource loading progress P calculation formula: P = (N c / N t ) × 100%, where N c is the number of loaded resources, and N t is the total number of resources.

[0062] Third, the gesture tracking algorithm is based on xr-ar-tracker. Hand feature extraction is performed through the key point set K = {P0, P5, P4, P8, P 12 , P 16 , P 20}, where P i represents the three-dimensional tracking coordinates of the i-th key point of the hand. The tracking accuracy δ is optimized by the least squares method: min∑(P i ' - P i )2, where P i ' is the predicted position and P i is the actual position.

[0063] Fourth, the particle system adopts a two-layer design. The first layer uses a fixed-point emission equation: P(t) = P0, and the particle life L = ∞; the second layer uses a spherical emitter, and the butterfly motion equation: P(t) = P0 + vt + (1 / 2)gt2, where v ∈ [0.3, 0.5] is the velocity vector, g = 0.08 is the acceleration due to gravity, the emission angle θ ∈ [70°, 110°], and the particle transparency α decays with time: α(t) = α0(1 - t / L), α0 is the initial particle transparency, t is the time parameter, and L ∈ [1.2, 2.0] is the particle life.

[0064] Fifth, the 3D model rendering is achieved through the affine transformation matrix T': T' = S × R × T, where S is the scaling matrix (scale ∈ [0.001, 0.4]), R is the rotation matrix (specified by the Euler angles θx, θy, θz), and T is the translation matrix. The lighting calculation adopts the Phong lighting model: I = ka × Ia + kd × Id × (N·L) + ks × Is × (R·V) n , where ka, kd, and ks are the ambient light, diffuse reflection, and specular reflection coefficients respectively, Ia, Id, and Is are the light source intensities, N is the normal vector, L is the light source direction, R is the reflection direction, V is the viewing direction, and n is the glossiness.

[0065] The AR face special effect interaction includes the following steps:

[0066] First, the system initializes the AR scene through the xr-scene component, adopts the AR system based on face recognition (modes: Face), and establishes a three-dimensional tracking coordinate system. The rendering pipeline uses WebGL technology and realizes spatial mapping through the composite transformation of the perspective projection matrix P and the view matrix V: M = P × V × T(x,y,z), where T(x,y,z) is the affine transformation matrix of the object, and the near clipping plane parameter is set to 0.01 to ensure the rendering accuracy at close range.

[0067] Second, the system implements a multi-level particle system (ParticleSystem) rendering algorithm. The particle motion trajectory follows the spherical emission equation (butterfly motion equation): P(t) = P0 + vt + (1 / 2)gt2, where P0 is the initial position, v is the velocity vector (range [0.1, 0.5]), and g is the gravitational acceleration (set to -9.8m / s 2 ). The particle lifetime L ∈ [0.5, 5.0] seconds, the emission angle θ ∈ [0°, 360°], and the number of particles is controlled by the capacity parameter (range [1, 300]).

[0068] Third, the face special effect positioning adopts an adaptive synchronization algorithm, and realizes the key point tracking of different special effects through the autoSyncMap mapping table. The synchronization parameter Si of the special effect type i is composed of the vector {a1, a2,..., a n}, where ai ∈ [-1, 46] represents the facial feature point index. The position matrix P = [xyz]T is transformed by the rotation matrix R'(α,β,γ) and the scaling matrix S(sx, sy, sz) to obtain the final rendering position: P' = R' × S × P.

[0069] Fourth, the lighting system adopts a multi-light source mixing algorithm, including ambient light, directional light, and spot light. The lighting intensity calculation uses an improved Phong lighting model: I = ka × Ia + kd × Id × (N·L) + ks × Is × (R·V) n , where ka, kd, and ks are the ambient light, diffuse reflection, and specular reflection coefficients respectively, N is the normal vector, L is the light source direction, R is the reflection direction, V is the viewing direction, and n is the glossiness.

[0070] Fifth, the 3D model rendering uses the GLTF format and realizes resource management through an asynchronous loading strategy. The model transformation matrix M = T × R' × S, where T is the translation matrix (controlled by the position parameter), R' is the Euler angle rotation matrix (rotation parameter, range [0°, 360°]), S is the scaling matrix (scale parameter, range [0.001, 0.8]), and the animation playback speed v is adjusted through the anim-speed parameter, range [0.1, 0.5], to ensure a smooth visual effect.

[0071] The AR full-body special effect interaction includes the following steps:

[0072] First, the system initializes the AR scene through the xr-scene component, sets the ar-system mode to "Body", and establishes a human body tracking coordinate system. Precise tracking of 9 key points of the human body is achieved through the auto-sync parameters "3, 4, 5, 6, 12, 13, 16, 17, 18", where the key point set K = {P3, P4, P5, P6, P 12 , P 13 , P 16 , P 17 , P 18} constitutes the basic skeleton model for human body pose estimation.

[0073] Second, the system implements a multi-level particle special effect rendering system. The particle emitter uses a point-shaped (PointShape) emission mode, and the particle motion state vector S = [P, v, L, α], where P is the position vector (-0.4 ≤ x ≤ 0.4, -0.1 ≤ y ≤ 0.45, z = 0), v is the velocity vector (v = 0 represents a static particle), L is the particle lifetime (L = 999999 represents continuous display), and α is the transparency (α ∈ [0, 1]). The particle size parameter size ∈ [0.1, 1.5], and stable single-particle rendering is achieved through capacity = 1 and emit-rate = 1.

[0074] Third, the model rendering uses the GLTF format and realizes spatial positioning through the affine transformation matrix T': T' = T(p) × R'(θ) × S(s), where T(p) is the translation transformation, p is the translation vector (px, py, pz) = (0, 0, 0.1), px = 0 means no offset in the X-axis direction, py = 0 means no offset in the Y-axis direction, and pz = 0.1 means a forward offset of 0.1 unit in the Z-axis direction, making the model slightly suspended. R'(θ) is the rotation transformation, θ is the Euler angle vector (θx, θy, θz) = (180°, 90°, 90°), θx = 180° means rotating 180 degrees around the X-axis to flip the model vertically, θy = 90° means rotating 90 degrees around the Y-axis to place the model laterally, and θz = 90° means rotating 90 degrees around the Z-axis to adjust the model's orientation. S(s) is the scaling transformation, s is the unified scaling factor, with a range of [0.0008, 0.2]. 0.0008 is used for the fine display of details of small objects such as butterflies, and 0.2 is used for the overall presentation of larger objects such as scene decorations. The scaling factor can be dynamically adjusted according to different models and display requirements. The model animation speed vector v is adjusted through the anim-speed parameter, v ∈ [0.1, 2.0], and transparent blending rendering is achieved through states = "alphaMode:BLEND,renderQueue:3000".

[0075] Fourth, the scene lighting system adopts a multi-light source mixing algorithm, including ambient light (ambient, intensity Ia = 0.6), directional light (directional, intensity Id = 3.5), and point light (point, intensity Ip = 2.5). The lighting calculation uses an improved Phong model: I = ka × Ia + kd × Id × cos(θ) + ks × Is × (cos(α))n, where ka, kd, and ks are the material coefficients respectively, θ is the incident angle, α is the reflection angle, and n is the glossiness. The spotlight (spot) parameters are configured as: inner-cone-angle = 35°, outer-cone-angle = 50° to achieve a soft edge transition.

[0076] Finally, the system realizes special effect switching through an asynchronous resource loading strategy. The resource loading state S consists of the loading progress Pr and the ready state Re: S = {Pr, Re}, where Pr = (N c / N t ) × 100% represents the loading progress, N c is the number of loaded resources, and N t is the total number of resources. The special effect type is controlled by effectType ∈ [-2, 6], and each special effect corresponds to a unique resource identifier ID = f(type) to ensure the precise loading and release of special effect resources.

[0077] AR image special effect interaction includes the following steps:

[0078] First, the system initializes the AR scene through the xr-scene component, sets the ar-system mode to "Marker", and establishes a three-dimensional tracking coordinate system based on image recognition. Feature point matching algorithm is used for image recognition, and the feature vector F = {f1, f2,..., f n} is extracted by SIFT (Scale-Invariant Feature Transform), where each feature point fi contains the position coordinates (x, y) and a 128-dimensional descriptor vector d.

[0079] Second, the system adopts a hierarchical rendering strategy to achieve a semi-transparent overlay effect through the watermark texture. The blending equation for the watermark layer: C = αS + (1 - α)D, where S is the source color, D is the destination color, and α is the transparency. The particle system parameters are configured as: capacity = 1 (capacity), emit-rate = 1 (emission rate), life-time = 999999 (particle life) to ensure the stable display of the watermark.

[0080] Third, the 3D model animation system adopts the keyframe interpolation algorithm. For time t ∈ [0, 1], the position vector P(t) is calculated by cubic spline interpolation: P(t) = ∑Bi(t)Pi, where Bi(t) is the basis function and Pi is the control point. The animation speed is adjusted through the anim-speed parameter, and transparent blending rendering is achieved through states = "alphaMode:BLEND,renderQueue:3000".

[0081] Fourth, the scene lighting system adopts a multi-light source mixing algorithm. The main light source configuration includes ambient light (ambient, intensity Ia = 1.0) and directional light (directional, intensity Id = 3.0). The lighting calculation uses an improved Blinn-Phong model: I = ka × Ia + kd × Id × (N · L) + ks × Is × (H · N)n, where H is the half-angle vector, H = (L + V) / |L + V|, N is the normal vector, L is the light source direction, V is the viewing direction, and n is the specular parameter.

[0082] Finally, the system implements a dynamic resource loading mechanism based on the effectType parameter. The resource identifier is generated through the mapping function f(t): f(t) = "butterfly" + t, t ∈ [0, 8]. The loading state transition satisfies the Markov chain property, and the state transition probability matrix P = [pi j ensures the reliability of resource loading, where pi j represents the probability of transitioning from state i to state j.

[0083] User interaction (implementing special effect switching, message interaction, barrage interaction, taking photos, recording, and sharing) includes the following steps:

[0084] First, the system implements a special effect switching mechanism, controlling the special effect type through the state transition function S(t): S(t) = mod(S(t - 1)+ΔS, N), where N is the total number of special effects, and ΔS is the state increment. Each special effect corresponds to a unique identifier ID = f(type), and the one-to-one correspondence between the special effect type and resources is achieved through the mapping table M = {mi j}.

[0085] Second, the message interaction system adopts a distributed storage architecture, and the message data structure D = {id, content, timestamp, position(x,y,z)}. The spatial positioning algorithm is implemented through ray casting: R(t) = O + tD, where O is the starting point of the ray, D is the direction vector, and t is a parameter. The message layout adopts a force-directed algorithm, and the repulsive force calculation formula: F = k / d 2 , where k is the elastic coefficient and d is the message spacing.

[0086] Third, the barrage system implements an algorithm based on time and space scheduling. The barrage motion equation: P(t)' = P0' + v't';

[0087] where, P(t)' is the current position vector of the barrage, P0' is the initial position vector of the barrage, v' is the velocity vector of the barrage, and t' is the barrage movement time. The collision detection adopts an AABB bounding box: overlap(A,B) = (|Ax - Bx| ≤ (Aw + Bw) / 2) ∧ (|Ay - By| ≤ (Ah + Bh) / 2). Where, A and B are two barrage objects to be detected, Ax and Bx are the x coordinates of the centers of barrages A and B, Ay and By are the y coordinates of the centers of barrages A and B, Aw and Bw are the widths of barrages A and B, Ah and Bh are the heights of barrages A and B, |Ax - Bx| is the distance between the centers of the two barrages in the x-axis direction, |Ay - By| is the distance between the centers of the two barrages in the y-axis direction, and ∧ is the logical AND operator, indicating that both conditions are satisfied. The barrage density control is through the Poisson distribution model: P(X = k) = (λke -λ ) / k!, where, P(X = k) is the probability of k barrages appearing in a certain time and space region, k is the number of barrages (non-negative integer), λ is the average density parameter, representing the expected number of barrages in the unit time and space region, e is the base of the natural logarithm (about 2.71828), k! is the factorial of k, λk is the kth power of λ, e -λ is the -λ power of e.

[0088] Fourth, the camera function realizes real-time image capture through frame buffer technology. The image quality parameter Q is determined by the resolution R and the compression rate C: Q = f(R, C), where R = width × height × dpi and C ∈ [0, 1]. Image enhancement adopts adaptive histogram equalization: p(i) = T(r) / n, where T(r) is the cumulative distribution function of the gray value r and n is the total number of pixels.

[0089] Fifth, the recording function adopts a multi-thread parallel processing architecture. The video encoding parameters include: frame rate fps = 30, bit rate br = 1000 kbps, resolution res = (w, h). The relationship between the recording time t and the file size S: S = (br × t) / 8 + O, where O is the file header overhead. The timer update frequency f = 1 Hz and the timing accuracy δt = ±0.1 s.

[0090] Sixth, the sharing function is implemented based on an asynchronous communication model. The data transmission rate R satisfies Shannon's theorem: R ≤ B × log(1 + S / N), where B is the bandwidth and S / N is the signal-to-noise ratio. The sharing state transition diagram G = (V, E), the vertex set V represents the set of states, and the edge set E represents the possible state transitions. The success rate P is optimized by the exponential backoff algorithm: t_retry = min(t_base × 2^n, t_max), where n is the number of retry times.

[0091] Finally, the system implements an intelligent recommendation algorithm based on user behavior. The user preference vector U = [u1, u2,..., u n , the special effect feature vector E = [e1, e2,..., e n , and the similarity calculation uses cosine similarity: sim(U, E) = (U · E) / (|U| × |E|). The recommendation priority P = α × sim + β × pop + γ × time, where pop is the popularity, time is the timeliness, and α, β, γ are weight coefficients and satisfy α + β + γ = 1.

[0092] User data storage includes the following steps:

[0093] First, the system adopts a distributed data storage architecture and establishes a user feedback data model D = {U, T, R, O}, where U represents the user basic information vector (age, gender, experience), T represents the task completion time matrix [t1, t2, t3, t4], R represents the scoring data matrix, and O represents the open answer set. Data synchronization adopts an incremental update strategy, and the update time interval Δt satisfies: Δt = min(t_base × 2^n, t_max), where n is the number of retry times.

[0094] Secondly, the questionnaire scoring system adopts a multi-dimensional evaluation model. The scoring dimensions include functionality F, learnability L, immersion I, comfort C, and satisfaction S. The weight coefficients for each dimension are w1, w2, w3, w4, and w5 respectively, and they satisfy ∑wi = 1. The comprehensive scoring calculation formula: Score = ∑(wi × Di), where Di represents the average score of each dimension, and the score range is [1, 5].

[0095] Thirdly, the system implements an asynchronous storage mechanism based on a cloud database. Data submission adopts a transaction processing model T = {op1, op2,..., op n}, and the transaction success rate P is optimized through an exponential backoff algorithm: t_retry = min(t_base × 2n, t_max). Data integrity is ensured through a verification function V(D): V(D) = ∏(vi), where vi ∈ {0, 1} represents the validity of each field.

[0096] Fourthly, the user feedback analysis adopts a multi-level clustering algorithm. For the scoring dataset R, the class center is calculated through the K-means algorithm: C = argmin∑∑||xi - μ||2, where xi is the sample point and μ is the clustering center. User clustering is evaluated through the silhouette coefficient S(i): S(i) = (b(i) - a(i)) / max{a(i), b(i)}, where a(i) is the average intra-group distance and b(i) is the average distance to the nearest neighbor group.

[0097] Fifthly, open-ended questions are analyzed using natural language processing techniques. Text vectorization adopts the TF-IDF algorithm: TF-IDF(t, d) = tf(t, d) × log(N / df(t)), where tf(t, d) is the term frequency, df(t) is the number of documents containing the word t, and N is the total number of documents. Sentiment analysis is calculated through the polarity score P: P = (P + - P - ) / (P + + P - ), where P + , P - are the weights and of positive and negative words respectively.

[0098] Sixthly, the system optimization strategy is based on the time series analysis of user feedback. The performance metric improvement rate R(t) is calculated: R(t) = (P(t) - P(t - 1)) / P(t - 1), where P(t) is the performance metric at time t and P(t - 1) is the performance metric at time t - 1. The user satisfaction trend is analyzed through a moving average algorithm: MA(n) = (1 / n)∑S(t - i), i ∈ [0, n - 1], where n is the time window size, S(t - i) is the user satisfaction score, and MA(n) is the moving average value.

[0099] The implementation of the AR function module of the present invention is as follows:

[0100] AR Gesture Special Effect Sub-module: The user develops an algorithm (mini-program) for processing user gesture inputs, starts the mini-program, scans and initializes the virtual butterfly model through the camera, guides and controls the flight or stay of the virtual butterfly through waving or clicking gestures, adjusts the rotation, zooming in or out of the butterfly model through gestures, the system tracks the user's actions, and provides real-time feedback according to the gestures. The user can select functions such as butterfly special effects, taking pictures, commenting, recording, etc. to interact and enhance the immersion of the interactive experience. The specific steps and interaction effects are shown in Figure 4 and Figure 5 respectively.

[0101] AR Face Special Effect Sub-module: The user starts the mini-program, scans the face through the camera, the system recognizes the user's facial features, and the system dynamically adds special effects such as butterfly wings or flower wreaths to the user's face to create an interactive effect. The face special effect tracks the user's facial movements in real time, making the butterfly fly around the user's face to increase the fun. The user can adjust the position and direction of the butterfly special effect by moving the face or adjusting the angle. The user can take pictures, record videos or share the special effect to social platforms to record and share the interactive experience. Through face recognition technology, butterfly wings or flower wreaths are dynamically added to the user's face to enhance the fun. The specific steps and interaction effects are shown in Figure 6 and Figure 7 respectively.

[0102] AR Whole Body Special Effect Sub-module: The user starts the mini-program, the camera captures the user's whole body, the system recognizes the human body contour and initializes the butterfly special effect, the system generates a dynamic butterfly special effect, making the user seem to be in a butterfly garden and flying around the user's body. Through whole body recognition technology, the butterfly special effect adjusts with the user's actions to achieve dynamic interaction. The user can make the butterfly fly around a specific part or stay by moving, rotating or stretching the body. The user can select special effects, take pictures, record videos and share them to social platforms to enhance the interactive experience. This process can achieve the special effect of butterflies flying around the user and improve the user's immersive experience. The specific steps and interaction effects are shown in Figure 8 and Figure 9 respectively.

[0103] AR Image Special Effect Sub-module: When the user launches the mini-program, the camera scans the images or objects in the environment. The system identifies and prepares to overlay the butterfly special effect. The system superimposes the butterfly special effect on the recognized image, and the butterfly interacts with the real environment to create a dreamy effect. The user can adjust the display effect of the butterfly special effect by moving or changing the perspective to enhance the interactivity. The system adjusts the position and behavior of the butterfly in real time according to environmental changes or user actions to closely integrate it with the surrounding environment. The user can take pictures, record videos, or save and share the special effect on social platforms to record and display the interactive experience. This process superimposes virtual butterflies onto the user's real-world view to create a dreamy visual effect. For the specific steps and interaction effects, please refer to Figure 10 and Figure 11 。

[0104] The operation process of the user message interaction function of the present invention is as shown in Figure 12 : The specific steps are as follows: The user clicks the comment button on the mini-program interface to enter the message interaction mode; the user enters the comment content in the text box and clicks the submit button to send the message; the submitted message will be displayed on the interface, and the user can view the comments of others; after the user selects the "open bullet screen" button, the comments will scroll on the screen in the form of bullet screens to enhance the interactive atmosphere; the user can reply to others' messages or interact through bullet screens to share and feedback the experience in real time. Figure 13 It is the effect diagram of the user interaction function.

[0105] Implementation of the gesture recognition module.

[0106] Gesture detection algorithm: Implement an algorithm that can capture and recognize user gesture actions in real time, such as waving and clicking.

[0107] Mapping from gesture to butterfly: Convert the recognized gesture into corresponding actions of the virtual butterfly, such as flying or staying on the user's palm.

[0108] The specific steps of gesture recognition are as follows:

[0109] First, the system adopts a multi-level gesture recognition framework and constructs a hand key point detection network through a deep learning model. After the input image I is feature-extracted, the feature map F = CNN(I) is obtained, and the key point set K = {P1, P2,..., P 21} is obtained through the regression network, where each key point Pi = (x, y, z, c) contains three-dimensional tracking coordinates and confidence. The key point detection accuracy is optimized by the mean square error loss function: L = (1 / n)∑||Pi' - Pi||2, where Pi' is the predicted position and Pi is the real position.

[0110] Secondly, gesture classification adopts a time-series feature analysis algorithm. The continuous frame features are captured by sliding the time window W(t) = {f(t - k),..., f(t)}, where f(t) is the gesture feature vector at the current frame t, f(t - k) is the gesture feature vector k frames before, k is the size of the time window, and W(t) is the sliding time window. The gesture state vector G = [θ1, θ 2, ..., θ n is calculated through the relative angles between key points: θi = arccos((P1P2 · P2P3) / (|P1P2| · |P2P3|)).

[0111] Among them, G = [θ1, θ 2, ..., θ n is the gesture state vector, θi is the bending angle of the i-th finger joint, n is the total number of angle features, P1, P2, and P3 are the 3D coordinates of the hand key points, P1P2 is the vector from P1 to P2, P2P3 is the vector from P2 to P3, P1P2 · P2P3 is the dot product of the two vectors, |P1P2| is the modulus of the vector P1P2, |P2P3| is the modulus of the vector P2P3, arccos is the inverse cosine function, and the included angle is calculated.

[0112] The gesture recognition accuracy is optimized by a softmax classifier: P(y|G) = exp(Wy · G) / ∑exp(Wj · G). Among them, P(y|G) is the probability of belonging to class y when the given gesture state vector is G, G is the input gesture state vector, y is a specific gesture class (such as "wave", "click"), Wy is the weight vector corresponding to class y, Wj is the weight vector of all classes j, exp is the natural exponential function, Wy · G is the dot product of the weight vector and the input vector, and ∑ is the sum over all possible classes j.

[0113] Thirdly, a gesture interaction dynamics model based on a physics engine is implemented. The butterfly motion equation: P(t) = P0 + vt + (1 / 2)at2, where v is the butterfly velocity vector, a is the butterfly acceleration vector. P0 is the initial position vector of the butterfly, t is the butterfly motion time parameter, and P(t) is the current position vector of the butterfly. a is determined by the resultant force: a = Ftotal / m, and the resultant force Ftotal includes three components: Ftotal = F + Fdrag + Frandom, F is the palm guiding force: F = k / d 2·(Phand - P) / |Phand - P|, where k is the gravitational coefficient in the range of [0.1, 1.0], d is the distance to the palm center, Phand is the palm position vector, and P is the current position vector of the butterfly. Fdrag is the air resistance: Fdrag = -βv2, where β is the drag coefficient 0.1 and v is the velocity vector. Frandom is the random perturbation force: |Frandom| ≤ 0.05|F|. The flapping frequency f of the butterfly wings is coupled with the motion state: f = f0 + α·|v| + γ·|a|, where f0 is the base frequency 4Hz, α is the velocity adjustment coefficient 0.5Hz / (m / s), and γ is the acceleration adjustment coefficient 0.2Hz / (m / s 2 ). |v| and |a| are the magnitudes of velocity and acceleration respectively. The palm guiding force F affects the acceleration a, thus affecting the velocity v and position P, and at the same time coupling to control the flapping frequency f of the wings, forming a complete physical motion system. When the palm moves, the butterfly will make natural acceleration, deceleration and turning movements under the action of the guiding force, driving the corresponding changes in the wing flapping. The flapping frequency f of the butterfly wings changes with the velocity v: f = f0 + α·|v|, where f0 is the base frequency and α is the adjustment coefficient.

[0114] Fourth, the system uses the state machine design pattern to realize the mapping from gestures to actions. The motion state transition matrix T = [ti j defines the transition probabilities between different gesture states, where ti j represents the probability of transitioning from state i to state j. The action smoothness is achieved through an interpolation function: M(t) = M1(1 - t) + M2t, where t ∈ [0, 1], and M1, M2 are the motion parameters of adjacent key frames.

[0115] Fifth, an adaptive gesture tracking algorithm is implemented. The hand motion trajectory is predicted by a Kalman filter: X(t|t - 1) = AX(t - 1) + BU(t) + w(t), where X(t) is the motion state vector [position p, velocity v, acceleration a]

[0116] , A is the motion state transition matrix describing the kinematic relationship, X(t|t - 1) is the motion prediction state vector, X(t - 1) is the motion history state vector, U(t) is the motion control input vector, B is the motion control matrix, w(t) is the motion process noise, which follows a Gaussian distribution N(0, Q), and t represents the time parameter. The coupling point lies in the palm guiding force: F = k / d 2·(Phand - P) / |Phand - P|, where Phand comes from the predicted hand position X(t|t - 1) of the Kalman filter, k is the gravitational coefficient [0.1, 1.0], and d is the distance from the butterfly to the predicted palm position. The hand trajectory X(t) predicted by the Kalman filter directly affects the acceleration a of the butterfly through the guiding force F, and then affects the velocity v and position P, while coupling to control the wing flapping frequency f.

[0117] Through the immersive AR technology, the present invention creates an interactive experience of blending virtual and real, enabling users to be in the dreamy world of butterflies and enjoy a fantastic immersive journey. The system includes AR gesture special effects, AR face special effects, AR full - body special effects, and AR image special effects, which are implemented through a WeChat mini - program, allowing users to experience at any time through their mobile phones.

[0118] The present invention uses AR technology to provide a more intuitive and vivid display method, enhancing the interaction between users and virtual butterflies through various interaction means such as gesture recognition, face recognition, full - body recognition, and image recognition. Users can guide the butterfly to fly or stay through gestures such as waving and clicking, and can also see the butterfly flying around themselves through face special effects and full - body special effects, as if being in a butterfly garden.

[0119] The present invention also provides an AR interaction system, including:

[0120] A key - point extraction module, used to initialize the AR scene, extract multiple key points of the object to be interacted with, and establish a three - dimensional tracking coordinate of the object to be interacted with according to the multiple key points;

[0121] A motion - state transition - matrix acquisition module, used to construct a motion - state transition matrix representing different states of the object to be interacted with according to the three - dimensional tracking coordinate of the object to be interacted with;

[0122] A motion - trajectory prediction - equation acquisition module, used to obtain a motion - trajectory prediction equation of the object to be interacted with based on the motion - state transition matrix and in combination with a Kalman filter;

[0123] A butterfly - motion - equation construction module, used to obtain the velocity vector and acceleration vector of butterfly motion, and construct a butterfly - motion equation according to the velocity vector and acceleration vector;

[0124] An interaction module, used to obtain the motion - state vector of the object to be interacted with and the position vector of the butterfly, input the motion - state vector of the object to be interacted with and the position vector of the butterfly into the motion - trajectory prediction equation of the object to be interacted with and the butterfly - motion equation respectively, and perform AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion - trajectory prediction equation and the butterfly - motion equation.

[0125] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the AR interaction method.

[0126] The present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor to execute the AR interaction method.

[0127] The above embodiments are only preferred specific embodiments of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of technical solutions that can be obviously obtained by those skilled in the art within the technical scope disclosed by the present invention all belong to the protection scope of the present invention.

Claims

1. An AR interaction method, characterized in that, It includes the following steps: Initialize the AR scene, extract multiple key points of the object to be interacted with, and establish the three-dimensional tracking coordinates of the object to be interacted with based on the multiple key points; Construct a motion state transition matrix representing different states of the object to be interacted with according to the three-dimensional tracking coordinates of the object to be interacted with; Based on the motion state transition matrix and combined with the Kalman filter, obtain the motion trajectory prediction equation of the object to be interacted with; Obtain the velocity vector and acceleration vector of the butterfly's motion, and construct the butterfly motion equation according to the velocity vector and acceleration vector; Obtain the motion state vector of the object to be interacted with and the position vector of the butterfly, input the motion state vector of the object to be interacted with and the position vector of the butterfly into the motion trajectory prediction equation of the object to be interacted with and the butterfly motion equation respectively, and perform AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly motion equation.

2. The AR interaction method according to claim 1, wherein The object to be interacted with includes gestures, faces, and full-body postures. Multiple key points of gestures are extracted by a 21-point hand key point model; multiple key points of faces are extracted using a 68-point feature point model; multiple key points of full-body postures are extracted based on a 17-point skeleton model of the COCO standard.

3. The AR interaction method according to claim 1, wherein The butterfly motion equation is specifically as follows: P(t) = P0 + vt + (1 / 2)at2; Where, v is the butterfly velocity vector, a is the butterfly acceleration vector, P0 is the butterfly initial position vector, t is the butterfly motion time parameter, and P(t) is the butterfly current position vector.

4. The AR interaction method according to claim 1, wherein The motion trajectory prediction equation of the object to be interacted with is specifically as follows: X(t|t - 1) = AX(t - 1) + BU(t) + w(t); Where, A is the motion state transition matrix, B is the motion control matrix, w(t) is the motion process noise, X is the motion state vector, t represents the time parameter, X(t|t - 1) is the motion prediction state vector, X(t - 1) is the motion historical state vector, and U(t) is the motion control input vector.

5. The AR interaction method according to claim 1, characterized in that After performing AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly motion equation, it further includes performing barrage interaction using a time and space scheduling algorithm, which is specifically implemented through a barrage motion equation. The barrage motion equation is specifically as follows: P(t)' = P0' + v't'; Where, P(t)' is the barrage current position vector, P0' is the barrage initial position vector, v' is the barrage velocity vector, and t' is the barrage motion time.

6. An AR interaction system, characterized in that, It includes: A key point extraction module, which is used to initialize the AR scene, extract multiple key points of the object to be interacted with, and establish the three-dimensional tracking coordinates of the object to be interacted with based on the multiple key points; A motion state transition matrix acquisition module, which is used to construct a motion state transition matrix representing different states of the object to be interacted with according to the three-dimensional tracking coordinates of the object to be interacted with; A motion trajectory prediction equation acquisition module, which is used to obtain the motion trajectory prediction equation of the object to be interacted with based on the motion state transition matrix and combined with the Kalman filter; A butterfly motion equation construction module, which is used to obtain the velocity vector and acceleration vector of the butterfly's motion, and construct the butterfly motion equation according to the velocity vector and acceleration vector; An interaction module, configured to obtain the motion state vector of an object to be interacted with and the position vector of a butterfly, input the motion state vector of the object to be interacted with and the position vector of the butterfly into the motion trajectory prediction equation of the object to be interacted with and the butterfly motion equation respectively, and perform AR interaction between the object to be interacted with and the butterfly according to the calculation results of the motion trajectory prediction equation and the butterfly motion equation.

7. A computer device, characterized in that, It includes a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the AR interaction method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the AR interaction method according to any one of claims 1-5.

Images