A virtual digital figure interaction method and system for accurate playing of advertisements

Abstract

The application discloses a virtual digital image interaction method and system for accurate advertisement playing, relates to the technical field of virtual interaction, and comprises the following steps: collecting original points of a user, generating a virtual avatar of the user by using a virtual avatar generation model, embedding the virtual avatar into a meta-universe virtual space, and binding a collision body; embedding a 3D advertisement into the meta-universe virtual space, binding the collision body, and performing outer expansion on the collision body to generate an interaction trigger shell; acquiring original motion frames of each part of the virtual avatar in a sliding window in the meta-universe virtual space, outputting the motion of the virtual avatar by constructing a motion recognition model; and constructing a motion response mapping table, so that the virtual avatar performs virtual interaction with the 3D advertisement in the meta-universe virtual space. The application adopts a space-time motion graph convolution network to fuse multi-source features such as bone vectors, speeds and accelerations, so as to improve the recognition capability of short-time motion and subtle interaction.

Description

Technical Field

[0001] This invention relates to the field of virtual interaction technology, and in particular to a virtual digital avatar interaction method and system for precise advertising playback. Background Technology

[0002] In recent years, virtual avatar-driven and immersive interaction technologies have rapidly developed in fields such as metaverse, immersive advertising, and remote collaboration. Traditional human 3D reconstruction and driving methods mostly rely on multi-camera arrays and triangulation principles. They reconstruct the human skeleton and surface details by simultaneously acquiring multi-view images, and then combine them with temporal models such as RNN / LSTM for motion prediction. These methods can achieve high accuracy in laboratory environments, but in practical deployments, they often face problems such as high equipment costs, complex installation and calibration, and the accumulation of synchronization errors. Moreover, multi-camera systems are sensitive to conditions such as lighting and background occlusion, limiting their widespread application in dynamic or confined spaces. In recent years, the 3D point cloud acquisition capabilities of single LiDAR sensors have been continuously improving, providing a new technical path for low-cost, real-time human 3D modeling and driving.

[0003] However, existing LiDAR-based methods for generating and driving virtual avatars still have limitations. On the one hand, point cloud data is susceptible to background interference and noise, lacking efficient background removal, denoising, and normalization mechanisms. This results in poor geometric consistency and orientation uniformity of virtual avatars under different acquisition environments, affecting the accuracy of subsequent interactions. On the other hand, the action recognition process largely relies on two-dimensional coordinates or joint angle sequences, failing to effectively integrate multi-dimensional dynamic features such as bone vectors, velocity, and acceleration, leading to insufficient accuracy in recognizing short-term, subtle movements. Furthermore, existing interaction detection methods are mostly based on positional contact judgments, lacking a semantic mapping mechanism for actions triggered by virtual advertisements, thus failing to achieve complex interaction effects that conform to the advertising design logic. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a virtual digital avatar interaction method for precise ad playback, which solves the problems of low modeling accuracy and unstable action recognition in the interaction between virtual avatars and metaverse ads.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a virtual digital avatar interaction method for precise ad playback, which includes: removing background point cloud from the original point cloud of the user to obtain a variable point cloud; performing spatial clustering and principal component analysis on the variable point cloud to select the user's human body point cloud; generating the user's virtual avatar based on a continuous human body point cloud sequence using a virtual avatar generation model; embedding the virtual avatar into the metaverse virtual space and binding a collision body.

[0008] Embed 3D ads into the metaverse virtual space, bind them to colliders, and expand the colliders of the 3D ads to generate interactive trigger shells.

[0009] The original motion frames of each part of the virtual avatar in the metaverse virtual space within the sliding window are obtained, a spatiotemporal motion graph is constructed, and an action recognition model is constructed by using a spatiotemporal graph-based convolutional network. The spatiotemporal motion graph is then stacked with three convolutions to output the virtual avatar's actions.

[0010] By detecting the interaction triggers of the virtual avatar with the 3D advertisement's trigger shell, virtual interaction events are determined, and by constructing an action response mapping table, the virtual avatar interacts virtually with the 3D advertisement in the metaverse virtual space.

[0011] As a preferred embodiment of the virtual digital avatar interaction method for precise ad playback described in this invention, the specific steps for removing background point clouds from the original point clouds collected from users are as follows:

[0012] Before the user enters the point cloud acquisition area, several frames of background point cloud in the point cloud acquisition area are continuously acquired, and the three-dimensional coordinates of all points are averaged to obtain the static scene point cloud.

[0013] Calculate the Euclidean distance between each point in the original point cloud and all points in the static scene point cloud;

[0014] Based on the sampling accuracy of the point cloud acquisition device, a Euclidean distance threshold is set;

[0015] If the minimum Euclidean distance of each point in the original point cloud of each frame is greater than the Euclidean distance threshold, it is determined to be a changed point and retained; otherwise, it is determined to be a fixed point and discarded.

[0016] As a preferred embodiment of the virtual digital avatar interaction method for precise advertising playback described in this invention, the specific steps for filtering the user's human body point cloud from the changing point cloud are as follows:

[0017] The variable point cloud is subjected to noise reduction and normalization processing;

[0018] Using Euclidean clustering, spatial clustering is performed on the noise-reduced and normalized variation point cloud to automatically identify and filter out the user's human body backbone point cloud.

[0019] Principal component analysis was performed on the remaining point cloud after spatial clustering to obtain the point cloud of the user's limbs;

[0020] Based on the spatial relationships of the human body, assemble the user's trunk point cloud and limb point cloud, and output the user's human body point cloud.

[0021] As a preferred embodiment of the virtual digital avatar interaction method for precise ad targeting described in this invention, the specific steps for generating the user's virtual avatar are as follows:

[0022] In the temporal feature encoding layer of the virtual avatar generation model, a multi-layer transform encoder sequentially performs self-attention operations on the human point cloud of each frame and the virtual avatar of the previous frame, and outputs a temporal feature vector.

[0023] In the CVAE encoding layer, based on two sets of fully connected layers, the temporal feature vectors are mapped to the latent space mean and standard deviation, respectively;

[0024] Based on the latent space mean and standard deviation, the latent variables are sampled using the reparameter method;

[0025] In the CVAE decoding layer, the virtual avatar of the previous frame, the latent variables of the current frame, and the temporal feature vector are passed through multiple fully connected layers and activation layers to output the user's virtual avatar.

[0026] As a preferred embodiment of the virtual digital avatar interaction method for precise ad playback described in this invention, the specific steps for binding a collider to a 3D ad embedded in the metaverse virtual space are as follows:

[0027] The bounding box size of the 3D mesh based on 3D advertising is calculated, and the aspect ratio of the bounding box is determined.

[0028] If the length, width, and height ratio of the bounding box meets the requirements of a regular shape, then a geometric prototype collider is used; otherwise, a convex hull collider is used.

[0029] As a preferred embodiment of the virtual digital avatar interaction method for precise advertising playback described in this invention, the specific steps for constructing the spatiotemporal motion graph are as follows:

[0030] Obtain the original motion frames of the virtual avatar within the sliding window;

[0031] Calculate joint attribute features based on the position vectors and rotation quaternions of each joint in each original motion frame;

[0032] Using joints as nodes and joint attribute features as node features, the connection relationship between parent and child joints in the same original action frame is a spatial edge, and for the same node, a temporal edge is established between adjacent frames.

[0033] Based on joint attribute features, spatial edge weights and temporal edge weights are calculated to complete the construction of the spatiotemporal motion graph.

[0034] As a preferred embodiment of the virtual digital avatar interaction method for precise advertising playback described in this invention, the specific steps for outputting the virtual avatar's actions are as follows:

[0035] Spatial adjacency matrix and temporal adjacency matrix are formed based on spatial edge weight and temporal edge weight respectively, and the degree matrix is ​​symmetrically normalized.

[0036] Using the node features of the spatiotemporal action graph, the symmetric normalized spatial adjacency matrix, and the temporal adjacency matrix as input, the action recognition model performs three-segment convolution stacking and global average pooling to obtain the action description vector.

[0037] The fully connected layer maps the log odds vector of each action category, and performs probability mapping on the log odds vector to obtain the predicted probability of each action category. The action category with the highest predicted probability is taken as the action of the virtual avatar.

[0038] As a preferred embodiment of the virtual digital avatar interaction method for precise ad playback described in this invention, the virtual interaction event refers to the event being triggered when the collider of the virtual avatar enters the range of the interaction trigger shell of the 3D ad, and the actions within the contact event occurrence time window are recorded.

[0039] As a preferred embodiment of the virtual digital avatar interaction method for precise advertising playback described in this invention, the specific steps for denoising and normalizing the changing point cloud are as follows:

[0040] The normal vector filtering method is used to remove noisy point clouds from the changing point cloud, and the moving least squares method is used to smooth the denoised changing point cloud.

[0041] The point cloud after noise reduction is decentralized, scaled, and oriented.

[0042] Secondly, the present invention provides a virtual digital avatar interactive system for precise advertising playback, comprising,

[0043] The virtual avatar generation module is used to remove the background point cloud from the original point cloud of the user to obtain the variable point cloud. The variable point cloud is then subjected to spatial clustering and principal component analysis to select the user's human body point cloud. Based on the continuous human body point cloud sequence, the virtual avatar generation model is used to generate the user's virtual avatar. The virtual avatar is then embedded into the metaverse virtual space and bound to a collision body.

[0044] The ad embedding module is used to embed 3D ads into the metaverse virtual space, bind colliders, and expand the colliders to generate interactive trigger shells.

[0045] The action recognition module is used to acquire the original action frames of various parts of the virtual avatar in the metaverse virtual space within the sliding window, construct a spatiotemporal action graph, and build an action recognition model through a spatiotemporal graph-based convolutional network. The spatiotemporal action graph is then subjected to three-segment convolution stacking to output the action of the virtual avatar.

[0046] The virtual interaction module is used to detect the interaction between the virtual avatar's trigger and the 3D advertisement's trigger shell, determine the virtual interaction event, and construct an action response mapping table so that the virtual avatar can interact with the 3D advertisement in the metaverse virtual space.

[0047] The beneficial effects of this invention are as follows: By combining the virtual avatar generation model of Transformer and CVAE, this invention realizes the generation and dynamic driving of virtual avatars under a single LiDAR input condition, reducing equipment and deployment costs. In the action recognition stage, a spatiotemporal action graph convolutional network is used to fuse multi-source features such as bone vectors, velocity and acceleration, thereby improving the recognition ability of short-term actions and subtle interactions. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart for a virtual digital avatar interaction method used for precise ad targeting.

[0050] Figure 2 A flowchart for removing background point clouds.

[0051] Figure 3 A flowchart for generating a virtual avatar.

[0052] Figure 4 A flowchart for constructing a spacetime action graph. Detailed Implementation

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0055] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0056] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a virtual digital avatar interaction method for precise advertising playback, comprising the following steps:

[0057] S1. Use a LiDAR sensor to collect the user's original point cloud. Remove the background from the original point cloud to obtain the variable point cloud. Perform spatial clustering and principal component analysis on the variable point cloud to filter out the user's human body point cloud. Based on the continuous human body point cloud sequence, use a virtual avatar generation model to generate the user's virtual avatar and embed it into the metaverse virtual space.

[0058] A LiDAR sensor is fixedly installed at a location on the top or side wall of the user's activity area that is unobstructed and free from reflections from obstructions. The LiDAR sensor is automatically triggered every 50ms (20Hz) to obtain the user's original point cloud.

[0059] Upon initial startup, before the user enters the point cloud acquisition area, several frames of background point cloud data from the acquisition area are continuously acquired. The 3D coordinates of all points in the background point cloud of all frames are averaged to obtain a static scene point cloud. A frame of original point cloud is acquired in real time, and the Euclidean distance between each point in the original point cloud and all points in the static scene point cloud is calculated. An Euclidean distance threshold is set based on the sampling accuracy of the LiDAR device. For example, if the sampling resolution of the LiDAR device is 1cm, the Euclidean distance threshold can be set to 1cm. When the minimum Euclidean distance of each point in each frame of original point cloud is greater than the Euclidean distance threshold, it is determined to be a changed point and retained. Otherwise, it is determined to be a fixed point and discarded to obtain a changed point cloud.

[0060] A combination of normal vector filtering and moving least squares is used to denoise the changed point cloud. The denoised changed point cloud is then normalized. The specific steps are as follows:

[0061] The noise reduction process for the changing point cloud includes noise removal and smoothing. Specifically, the normal vector filtering method is used. By calculating the normal vector of each point in the changing point cloud in the local neighborhood, the noise point cloud is identified and removed based on the difference between the normal vectors of each point in the changing point cloud and its neighbors. The moving least squares method is used to smooth the changing point cloud after removing the noise point cloud, thus completing the noise reduction of the changing point cloud.

[0062] By normalizing the denoised point cloud, including decentralization, scale normalization, and orientation normalization, the points of change under different acquisition environments are mapped to a unified spatial scale and reference coordinate system. Specifically, the geometric centroid coordinates of each frame of the point cloud are calculated, and the coordinates of all points in the point cloud are uniformly translated using the geometric centroid coordinates as the origin to obtain a decentralized point cloud. The maximum spatial expansion scale of the decentralized point cloud is calculated, such as the maximum Euclidean distance or the maximum axial length. A unified standard normalization scale is selected, such as a cube or sphere with a radius of 1 or a side length of 2, as the target range. Each point in the decentralized point cloud is scaled proportionally to obtain a scale-normalized point cloud. Principal component analysis is performed on the scale-normalized point cloud to solve for the scale. The covariance matrix of all points in the scale-normalized point cloud is obtained, and the principal axes and principal axis vectors of the scale-normalized point cloud are obtained. The principal axis vector represents the unit vector in the principal axis direction, including the first principal axis, the second principal axis, and the third principal axis. The first principal axis is the user's height direction, and the second and third principal axes are the user's width and thickness directions, respectively. The rotation axis and the included angle between the first principal axis and the world coordinate system Y-axis are calculated, and the rotation matrix is ​​constructed using the Rodriguez formula based on the rotation axis and the included angle. The rotation matrix is ​​used to unify the orientation of all points in the scale-normalized point cloud. The second and third principal axes are then used to perform secondary alignment with the world Z-axis or X-axis to ensure left and right consistency, such as facing the world Z-axis, avoiding mirroring or inversion, thus completing the normalization of the variable point cloud.

[0063] Based on the denoised and normalized point cloud, the user's trunk and limbs are extracted, and the user's human body point cloud is output. The specific steps are as follows:

[0064] Using Euclidean clustering, spatial clustering is performed on the noise-reduced and normalized variable point cloud to automatically identify and filter out the user's human body trunk point cloud. Specifically, based on the spatial resolution and point cloud density of the LiDAR device, the neighborhood radius and minimum number of neighborhood points of the Euclidean clustering method are set. For example, the neighborhood radius is 3cm and the minimum number of neighborhood points is 10. For each point in the variable point cloud, all points in the three-dimensional space with an Euclidean distance less than the neighborhood radius are searched to form a neighborhood. Neighborhoods with more points than the minimum number of neighborhood points are taken as core points and expanded into clusters. All points belonging to the same cluster are connected into clusters. The cluster with the most points and the most concentrated spatial distribution is selected as the trunk cluster to obtain the user's human body trunk point cloud, including the user's head and torso.

[0065] Based on the residual clusters after spatial clustering, the spatial centroid coordinates, volume, number of points, and spatial distance from the backbone cluster are calculated for each residual cluster. Residual clusters whose spatial distance from the backbone cluster is within the human body structure range (e.g., 20-70 cm) are identified as suspected limb clusters. Principal component analysis is performed on these suspected limb clusters to fit their spatial principal axes. The spatial principal axes are represented as straight lines passing through the spatial centroid coordinates, and their direction vectors are the eigenvectors corresponding to the largest coordinate axis eigenvalues. Specifically, each point in the suspected limb cluster is translated relative to its spatial centroid coordinates to obtain a decentralized point cloud. Based on this decentralized point cloud, the covariance matrix of the suspected limb clusters is calculated. The covariance matrix is ​​decomposed into coordinate axis directions to obtain three eigenvalues ​​and their corresponding eigenvectors. The maximum eigenvalue is used as the eigenvector corresponding to the maximum eigenvalue as the direction vector of the spatial principal axis of the suspected limb cluster. The ratio of the maximum eigenvalue to the eigenvalues ​​in the other two directions is calculated. Based on the geometric characteristics of the human limb point cloud (the point cloud of the human limb usually presents a slender distribution in three-dimensional space, that is, the spatial expansion along the principal axis is significantly greater than the two directions perpendicular to the principal axis), it is determined whether the spatial principal axis of the suspected limb cluster is a slender structure. If it is not a slender structure, it is filtered out as noise. If it is a slender structure, it is determined to be a limb cluster. Based on the point cloud density of the limb cluster, the minimum bounding sphere is fitted by the least squares method to locate each key point in the limb cluster, including each joint of the limb, as well as the toes and fingertips, to obtain the user's limb point cloud.

[0066] Using the identified trunk cluster and limb cluster, the user's trunk point cloud and limb point cloud are automatically assembled according to the spatial relationship of the human body, and the user's human body point cloud is output.

[0067] Based on a continuous sequence of human point clouds, a virtual avatar for the user is generated using a virtual avatar generation model. The specific steps are as follows:

[0068] The virtual avatar generation model includes a temporal feature encoding layer, a CVAE encoding layer, and a CVAE decoding layer.

[0069] Each frame of human point cloud and the virtual avatar of the previous frame in the human point cloud sequence are concatenated into the input sequence of the temporal feature coding layer. Self-attention operation is performed sequentially through a multi-layer transform encoder. Specifically, each layer includes a multi-head attention mechanism, a feedforward fully connected network, residual connections and normalization operations to effectively capture the motion trend and key point association of the human point cloud sequence and output a temporal feature vector.

[0070] In the initial input, the previous frame's virtual avatar is missing. An initial virtual avatar is constructed using static templates such as the standard human body T-pose or A-pose, which is then stitched together with the first frame's human body point cloud. Specifically, the T-pose or A-pose is translated to the centroid of the current user's human body point cloud and its orientation is adjusted to match the human body point cloud. Based on the ratio of the user's height and arm span to the T-pose or A-pose, each part of the T-pose or A-pose is proportionally scaled to obtain the initial virtual avatar.

[0071] In the CVAE encoding layer, based on two sets of fully connected layers, including a mean fully connected layer and a standard deviation fully connected layer, the temporal feature vectors are mapped to the latent space mean and standard deviation, respectively, as calculated below:

[0072]

[0073]

[0074] In the formula, The latent space mean, and These represent the weight matrices of the mean-based fully connected layer and the standard deviation fully connected layer, respectively. and These represent the bias terms of the mean-based fully connected layer and the standard deviation fully connected layer, respectively. For time series feature vectors, The hidden space standard deviation;

[0075] Based on the latent space mean and standard deviation, the latent variables are sampled using the reparameter method, and the calculation formula is as follows:

[0076]

[0077] In the formula, As a latent variable, For standard normally distributed random noise, This represents element-wise multiplication;

[0078] In the CVAE decoding layer, the virtual avatar of the previous frame, the latent variables of the current frame, and the temporal feature vector are passed through multiple fully connected layers and activation layers to output the user's virtual avatar, including the three-dimensional spatial coordinates and orientation of various parts of the human body, such as the head, hands, and feet.

[0079] It should be noted that the virtual avatar generation model used in this invention combines the Transformer encoder and the Conditional Variational Autoencoder (CVAE) in an end-to-end architecture. It is specifically designed for real-time driving scenarios of virtual avatars with a single LiDAR point cloud input. Compared with the conventional virtual avatar construction method of "multiple cameras + triangulation / RNN video estimation", the "single LiDAR + virtual avatar generation model" is more cost-effective. Only one LiDAR is needed to obtain the user's complete 3D point cloud, avoiding the calibration and synchronization complexity of multiple cameras. The Transformer's multi-head self-attention mechanism can focus on the information of any two time points in one forward propagation and capture the global trend of action changes, which is something that RNN / LSTM cannot achieve efficiently. In addition, the CVAE encoder can be directly extended to multimodal, such as LiDAR + RGB input, and the CVAE decoder can be adapted to different types of virtual avatars.

[0080] By constructing an affine transformation matrix, the virtual avatar is mapped to the global coordinate system of the metaverse virtual space. The calculation formula is as follows:

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] In the formula, Let be the affine transformation matrix. For scaling matrices, The rotation matrix is ​​calculated using Rodriguez's formula. It is a translation vector. It is the identity matrix. For the mapped rotation angle, Mapping rotation axis, Describes the antisymmetric matrix of the mapped rotation axis. The first principal axis vector of the virtual avatar, For example, the vertical unit vector in the metaverse virtual space, such as , The height of the virtual avatar is the same as the user's height. The target height set for the metaverse virtual space. The geometric centroid of the virtual avatar The initial embedding position for virtual avatars in the metaverse virtual space, such as the center point of the entrance to the advertising area;

[0088] Add colliders to various parts of the virtual avatar, such as rigid body components like Capsule, Box, and Sphere Collider, as well as triggers for detecting physical interactions with 3D ads in the metaverse virtual space.

[0089] S2. Embed the 3D advertisement into the metaverse virtual space, and generate an interactive trigger shell that is consistent with the shape of the 3D advertisement and expands outward by a fixed distance in all directions by binding a collider.

[0090] In the metaverse virtual space, fix several candidate advertising anchor points, such as walls or the ground, and mark the position vector, surface normal vector and minimum safe displacement of each candidate advertising anchor point;

[0091] By calculating the difference between the position vectors of the virtual avatar's eyes and the candidate ad anchor points, the viewing distance vector is obtained. The scaling factor of the 3D ad is calculated using the inverse distance scaling rule. Based on the frontal normal vector of the 3D ad and the surface normal vector of the ad anchor point, the rotation matrix of the 3D ad is calculated using the Rodriguez formula. Based on the minimum safe outward displacement, the 3D ad is moved outward along the direction of the surface normal vector to obtain the translation vector of the 3D ad. Before the ad is placed, ray detection can be used to confirm that there is no obstruction between the ad and the virtual avatar's line of sight. If obstruction is detected, it can be directly replaced with a backup anchor point.

[0092] Based on the homogeneous transformation matrix composed of 3D advertising scaling factor, 3D advertising rotation matrix and 3D advertising translation vector, 3D advertising is embedded into the metaverse virtual space.

[0093] To embed a collider into the 3D advertisement, an interactive trigger shell is generated that matches the shape of the 3D advertisement and extends outward by a fixed distance in all directions. Specifically, based on the bounding box size of the 3D advertisement (i.e., the maximum size along the X, Y, and Z axes in the world coordinate system), the aspect ratio of the bounding box is calculated. If the aspect ratio of the bounding box satisfies a regular shape, such as a cuboid, cylinder, or cube, a geometric prototype collider, such as a Box Collider or Capsule Collider, is used; otherwise, a convex hull collider, such as a Convex Mesh Collider, is used. For the interactive trigger shell of the geometric prototype collider, such as a Box Collider, the size vector of the Box Collider is extended outward based on the extension thickness in the global coordinate system of the metaverse virtual space. For the convex hull collider, such as a Convex Mesh Collider, the normal vector of each vertex is calculated, and each vertex is translated outward along the normal vector based on the extension thickness. The extension thickness is preset according to the size of the 3D advertisement, for example, it can be in the range of 0.2m to 0.5m.

[0094] S3. Obtain the original motion frames of each part of the virtual avatar in the sliding window in the metaverse virtual space. Take each joint of each original motion frame as a node and the direct connection between each node as a spatial edge. At the same node, establish temporal edges between adjacent frames to construct a spatiotemporal motion graph. Based on the spatiotemporal graph convolutional network, construct an action recognition model. Perform three-segment convolution stacking on the spatiotemporal motion graph to output the virtual avatar's action.

[0095] Obtain the original motion frames of each part of the virtual avatar in the metaverse virtual space within the sliding window, including the position vectors and rotation quaternions of each joint;

[0096] Based on the position vectors and rotation quaternions of each joint in each original motion frame, joint attribute features are calculated, including bone vectors, joint velocities, and joint accelerations. The bone vector is represented as a three-dimensional vector pointing from the parent joint to the child joint. For example, the bone vector of the forearm is represented as a three-dimensional vector pointing from the elbow joint to the wrist joint. The joint velocity includes the joint linear velocity and the joint angular velocity. Based on the joint rotation quaternions of two adjacent frames, the relative joint rotation quaternions are obtained, and the joint angular velocity is calculated using the quaternion logarithmic mapping method. The joint acceleration includes the joint linear acceleration and the joint angular acceleration.

[0097] It should be noted that simple coordinate sequences have limited ability to distinguish action categories, especially when there are large differences in the body proportions of different users, which can easily lead to confusion. By adding bone vectors to reflect the topological structure and capturing dynamic feature changes through velocity and acceleration, the discriminative ability of spatiotemporal graph convolution is enhanced, and short-term actions can be identified more accurately.

[0098] Based on bone vectors, joint velocities, and joint accelerations, a spatiotemporal motion graph is constructed and encoded using a spatiotemporal graph convolutional network to output motion categories. The specific steps are as follows:

[0099] Based on the position vectors and rotation quaternions of each joint, bone vectors, joint velocities, and joint accelerations are calculated to construct a spatiotemporal motion graph. Specifically, each joint in each frame of the virtual avatar is considered a node, with node features being joint attribute features. Joints that conform to anatomical adjacency relationships are connected sequentially to form an initial skeletal tree structure. Using each joint as a node, the direct connections between parent and child joints are used to obtain the avatar's skeleton topology. The direct connection relationship between parent and child joints within the same frame in the avatar's skeleton topology is used as a spatial edge. By jointly calculating bone vector differences, linear velocity differences, and angular velocity differences, the spatial edge weights are obtained, as shown in the following formula:

[0100]

[0101] In the formula, Indicates the parent node With child nodes Spatial edge weights, and These are the linear velocities of the parent and child nodes, respectively. and These are the angular velocities of the parent and child joints, respectively. Indicates the parent node Point to child node bone vector, Indicates the parent node Point to child node The prior unit bone vector originates from the parent node in the standard human T-pose or A-pose. Point to child node The bone vector, which is a unit vector, remains constant throughout the sliding window. , and These are the scale parameters for linear velocity difference, angular velocity difference, and bone vector change, respectively.

[0102] Furthermore, the scale parameters for linear velocity differences, angular velocity differences, and bone vector changes are obtained by using the parent node. With child nodes The pooling statistics within the sliding window are used to obtain the scale parameter of the linear velocity difference as an example. The parent node is calculated separately within the sliding window. With child nodes The sample variance of the linear velocity modulus sequence is calculated, and the scaling parameter of the linear velocity difference is calculated using the pooled standard deviation method. The calculation formula is as follows:

[0103]

[0104] In the formula, Indicates the parent node With child nodes The scale parameter of the linear velocity difference, and Parent nodes With child nodes The number of samples in the linear velocity modulus sequence. and Parent nodes With child nodes The sample variance of the linear velocity modulus sequence. Indicates the degrees of freedom to be merged;

[0105] For the same node, a temporal edge is established between adjacent frames. The calculation method for the spatial edge weight is based on the joint calculation of the node's linear velocity change, linear acceleration change, angular velocity change, and angular acceleration change to obtain the temporal edge weight.

[0106] Spatial and temporal adjacency matrices are formed based on spatial edge weights and temporal edge weights, respectively, and the degree matrix is ​​symmetrically normalized. Specifically, an initial spatial adjacency matrix of dimension K×K, empty, is constructed, where K represents the number of nodes in the spatiotemporal action graph. If the parent node... With child nodes If there is a direct connection in the virtual avatar skeleton topology, then the spatial edge weight is assigned to the corresponding position in the initial adjacency matrix; otherwise, it is 0, thus obtaining the spatial adjacency matrix. Construct an initial time adjacency matrix with dimension K×K and no space. Assign the time edge weight to the diagonal position to obtain the time adjacency matrix. Then, perform degree matrix symmetric normalization on the spatial adjacency matrix and the time adjacency matrix respectively.

[0107] It should be noted that although the temporal and spatial adjacency matrices can be constructed directly without skipping the spatiotemporal motion graph, constructing the spatiotemporal motion graph first can explicitly ensure that spatial edges conform to the skeletal topology and that temporal edges only connect adjacent frames of the same joint. This avoids semantic errors in the virtual avatar's actions from the source, ensuring that the subsequent interaction logic with 3D advertisements is realistic and reliable. In addition, nodes in the spatiotemporal motion graph can be directly bound to bone vectors, linear velocity, linear acceleration, angular velocity, and angular acceleration, so that the weight calculation of spatial and temporal edges does not need to repeatedly parse the original data, improving the efficiency and real-time performance of the temporal and spatial adjacency matrix generation.

[0108] Based on a spatiotemporal graph convolutional network, an action recognition model is constructed. The node features of the spatiotemporal action graph are subjected to three stacked convolutional segments. Each segment consists of a graph convolutional layer and a temporal convolutional layer in sequence. Channel compression is performed at the end of each segment to output the action of the virtual avatar. The specific steps are as follows:

[0109] Taking the node features, spatial adjacency matrix, and temporal adjacency matrix of the spatiotemporal action graph as input, in the first segment convolutional layer, the node features are weighted and aggregated in the spatial dimension using the spatial adjacency matrix. That is, the node features of each node are obtained by weighted summation of the features of the spatial neighbor nodes. The spatial adjacency matrix is ​​then used to perform a linear transformation to extract the spatial correlation features between nodes, outputting a first intermediate feature tensor. The temporal adjacency matrix is ​​then used to perform a one-dimensional convolution on the intermediate feature tensor in the temporal dimension to obtain a second intermediate feature tensor. This first tensor is then normalized and subjected to a non-linear activation function, such as the ReLU function, for non-linear feature mapping. This first tensor is then output as the input for the second convolution, and so on. After the second and third convolutions, three tensors are output and global average pooling is performed to obtain the action description vector. Through the mapping of the fully connected layer, the log odds vector of each action category is output. The log odds vector is then subjected to probability mapping to obtain the predicted probability of each action category. The action category with the highest predicted probability is taken as the action of the virtual avatar.

[0110] By continuously collecting virtual avatar action frame sequences within the metaverse virtual space and assigning action category labels such as click, wave, and grab, standard action frames with action labels are output. A standard spatiotemporal action graph and spatial and temporal adjacency matrices are constructed, and the action recognition model is iteratively trained. The predicted probability of the action category output by the action recognition model is compared with the action labels of the standard action frames, and a weighted cross-entropy loss is calculated. Based on the weighted cross-entropy loss value, the Adam optimization algorithm is used for backpropagation to update the weight matrix and bias terms of each segment in the action recognition model until the convergence condition of the action recognition model is met. For example, the weighted cross-entropy loss value does not decrease after ten consecutive training rounds.

[0111] S4. By detecting the interaction triggers of the virtual avatar and the interaction trigger shell of the 3D advertisement, the virtual interaction event is determined, and an action response mapping table is constructed. The virtual avatar interacts with the 3D advertisement in the metaverse virtual space.

[0112] In the metaverse virtual space, the virtual avatar triggers monitor in real time whether the virtual avatar's collider enters the interaction trigger shell of the 3D advertisement. When the virtual avatar's collider enters the interaction trigger shell of the 3D advertisement, a virtual interaction event is triggered, and the action category within the contact event time window is recorded.

[0113] Based on the design goals and interaction requirements of 3D advertising, define the possible response types that the advertising may perform, such as pop-up information: displaying detailed information about the advertising, such as text, images or videos; shape transformation: the advertising changes shape, color or animation effects, such as rotating, zooming in or out; follow movement: the advertising moves with the virtual avatar's hand or body, etc.

[0114] Under the condition of being triggered by a contact event, a one-to-one mapping relationship is set according to the action category and response type, and an action response mapping table is output. For example, if the action is click, the 3D advertisement will pop up detailed information; if it is wave, the 3D advertisement will be translated; if it is grab, the advertisement will be "picked up" and move with the virtual avatar.

[0115] This embodiment also provides a virtual digital avatar interaction system for precise ad playback, including: a virtual avatar generation module, used to remove background point clouds from the original point clouds of the collected users to obtain variable point clouds, perform spatial clustering and principal component analysis on the variable point clouds to select the user's human body point clouds, generate the user's virtual avatar based on a continuous sequence of human body point clouds using a virtual avatar generation model, embed the virtual avatar into the metaverse virtual space, and bind a collision body;

[0116] The ad embedding module is used to embed 3D ads into the metaverse virtual space, bind colliders, and expand the colliders to generate interactive trigger shells.

[0117] The action recognition module is used to acquire the original action frames of various parts of the virtual avatar in the metaverse virtual space within the sliding window, construct a spatiotemporal action graph, and build an action recognition model through a spatiotemporal graph-based convolutional network. The spatiotemporal action graph is then subjected to three-segment convolution stacking to output the action of the virtual avatar.

[0118] The virtual interaction module is used to detect the interaction between the virtual avatar's trigger and the 3D advertisement's trigger shell, determine the virtual interaction event, and construct an action response mapping table so that the virtual avatar can interact with the 3D advertisement in the metaverse virtual space.

[0119] This embodiment also provides a computer device applicable to the virtual digital avatar interaction method for precise advertising playback, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the virtual digital avatar interaction method for precise advertising playback as proposed in the above embodiment.

[0120] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0121] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the virtual digital image interaction method for precise advertising playback as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0122] In summary, this invention combines the Transformer and CVAE virtual avatar generation model to achieve the generation and dynamic driving of virtual avatars under a single LiDAR input condition, reducing equipment and deployment costs. In the action recognition stage, a spatiotemporal motion graph convolutional network is used to fuse multi-source features such as bone vectors, velocity, and acceleration to improve the recognition ability of short-term actions and subtle interactions.

[0123] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A virtual digital figure interaction method for accurate playing of advertisements, characterized in that: Comprise, The original point cloud of the user is removed from the background point cloud to obtain a change point cloud, the change point cloud is spatially clustered and principal component analyzed, the human point cloud of the user is screened out, a virtual avatar of the user is generated based on a continuous human point cloud sequence and a virtual avatar generation model, the virtual avatar is embedded in a meta-universe virtual space, and a collision body is bound; The virtual avatar of the user is generated, and the specific steps are as follows, In the time sequence feature coding layer of the virtual avatar generation model, self-attention operations are sequentially performed on each frame of human point cloud and the previous frame of virtual avatar through a multi-layer transformation encoder, and a time sequence feature vector is output; In the CVAE coding layer, the time sequence feature vector is mapped to the mean and standard deviation of the hidden space based on two groups of fully connected layers; Based on the mean and standard deviation of the hidden space, the reparameterization method is used to sample the hidden variable; In the CVAE decoding layer, the previous frame of virtual avatar, the current frame of hidden variable and time sequence feature vector are outputted through a multi-layer fully connected layer and an activation layer to output the virtual avatar of the user; The 3D advertisement is embedded in the meta-universe virtual space, the collision body is bound, and the collision body of the 3D advertisement is expanded to generate an interactive trigger shell; The original action frames of each part of the virtual avatar in the sliding window in the meta-universe virtual space are obtained, a space-time action graph is constructed, and an action recognition model is constructed based on a space-time graph convolution network, three convolution stacks are performed on the space-time action graph, and the action of the virtual avatar is outputted; The space-time action graph is constructed, and the specific steps are as follows, The original action frames of the virtual avatar in the sliding window are obtained; Based on the position vector and rotation quaternion of each joint in each frame of original action frame, the joint attribute feature is calculated; Taking the joint as a node and the joint attribute feature as a node feature, the connection relationship between parent and child joints in the same frame of original action frame is a space edge, and a time edge is established between adjacent frames for the same node; Based on the joint attribute feature, the space edge weight and the time edge weight are calculated, and the construction of the space-time action graph is completed; The action of the virtual avatar is outputted, and the specific steps are as follows, The space edge weight and the time edge weight are used to form a space adjacency matrix and a time adjacency matrix respectively, and the degree matrix is symmetrically normalized; The node feature of the space-time action graph, the symmetrically normalized space adjacency matrix and the time adjacency matrix are inputted, three convolution stacks are performed in the action recognition model, and global average pooling is performed to obtain an action description vector; The logit vector of each action category is outputted through the mapping of the fully connected layer, the probability of the logit vector is mapped to obtain the prediction probability of each action category, and the action category with the maximum prediction probability is taken as the action of the virtual avatar; By detecting the interaction between the trigger of the virtual avatar and the interactive trigger shell of the 3D advertisement, the virtual interaction event is judged, and the virtual avatar performs virtual interaction with the 3D advertisement in the meta-universe virtual space by constructing an action response mapping table.

2. The virtual digital character interaction method for advertisement precision play of claim 1, wherein: The original point cloud of the user is removed from the background point cloud, and the specific steps are as follows, Before the user enters the point cloud collection area, a plurality of frames of background point cloud of the point cloud collection area are continuously collected, and the three-dimensional coordinates of all points are averaged to obtain a static scene point cloud; Calculate the Euclidean distance of each point of the original point cloud from all points in the static scene point cloud; Set the Euclidean distance threshold based on the sampling accuracy of the point cloud acquisition device; When the minimum Euclidean distance of each point of each frame of the original point cloud is greater than the Euclidean distance threshold, it is determined as a changing point and retained, otherwise it is determined as a static point and removed.

3. The virtual digital character interaction method for advertisement precision play of claim 1, wherein: The user's human point cloud is filtered from the changing point cloud, and the specific steps are as follows, The changing point cloud is denoised and normalized; The Euclidean clustering method is used to perform spatial clustering on the denoised and normalized changing point cloud, automatically identify and filter out the user's human trunk point cloud; Based on the remaining point cloud after spatial clustering, principal component analysis is performed to obtain the user's limb point cloud; According to the spatial relationship of the human body, the user's trunk point cloud and limb point cloud are assembled to output the user's human point cloud.

4. The virtual digital character interaction method for advertisement precision play of claim 1, wherein: Bind the collision body to the 3D advertisement embedded in the meta-universe virtual space, and the specific steps are as follows, Based on the bounding box size of the three-dimensional grid of the 3D advertisement, calculate the bounding box length-width-height ratio; If the bounding box length-width-height ratio meets the regular shape, use the geometric prototype collision body, otherwise use the convex hull collision body.

5. The virtual digital character interaction method for advertisement precision play of claim 1, wherein: The virtual interaction event refers to when the collision body of the virtual avatar enters the range of the interaction trigger shell of the 3D advertisement, triggering the virtual interaction event, and recording the action within the contact event time window.

6. The virtual digital character interaction method for advertisement precision play of claim 3, wherein: The specific steps of denoising and normalizing the changing point cloud are as follows, Use the normal vector filtering method to remove noise points in the changing point cloud, and use the moving least squares method to smooth the denoised changing point cloud; Decentralize, scale normalize and orient normalize the denoised changing point cloud.

7. A virtual digital figure interaction system for accurate playing of advertisements, based on the virtual digital figure interaction method for accurate playing of advertisements according to any one of claims 1-6, characterized in that: It includes, The virtual avatar generation module is used to remove the background point cloud from the original point cloud of the user to obtain the changing point cloud, perform spatial clustering and principal component analysis on the changing point cloud, filter out the user's human point cloud, and based on the continuous human point cloud sequence, use the virtual avatar generation model to generate the user's virtual avatar, embed the virtual avatar in the meta-universe virtual space, and bind the collision body; The advertisement embedding module is used to embed the 3D advertisement into the meta-universe virtual space, bind the collision body, and expand the collision body to generate an interaction trigger shell; The action recognition module is used to obtain the original action frame of each part of the virtual avatar in the meta-universe virtual space within the sliding window, construct a spatio-temporal action graph, and construct an action recognition model based on the spatio-temporal graph convolution network, perform three convolution stacking on the spatio-temporal action graph, and output the action of the virtual avatar; The virtual interaction module is used to detect the trigger of the virtual avatar and the interaction trigger shell of the 3D advertisement, determine the virtual interaction event, and construct an action response mapping table, so that the virtual avatar can perform virtual interaction with the 3D advertisement in the meta-universe virtual space.

Images