A web front-end based interaction method, system and electronic device

Abstract

The application relates to a Web front-end-based interaction method, system and electronic equipment, which calls a terminal camera through a browser, acquires a video stream of a user, and processes the video stream at a browser front end to extract human key points; through a layered action mapping engine, the human key points are mapped into skeleton animation parameters of a virtual image; according to preset attributes of a virtual article and the skeleton animation parameters, the relative poses of the virtual article and the virtual image are calculated; in a browser environment, according to the relative poses, an interaction picture of the virtual article and the virtual image is rendered and displayed. The application solves the problems of high hardware dependency, poor cross-platform compatibility, simple interaction link, and serious delay and distortion in the prior art. Without special hardware, the application reduces the use threshold of users and realizes cross-platform compatibility based on a Web technology stack.

Description

Technical Field

[0001] This application relates to the field of virtual reality, and in particular to an interactive method, system, and electronic device based on a web front-end. Background Technology

[0002] With the development of internet technology, users' demands for immersive and realistic online interactions are increasing. Traditional virtual avatar interactions often rely on preset animations or simple clicks, failing to respond to users' natural movements in real time, resulting in a stiff and unrealistic interactive experience.

[0003] In existing technologies, methods for implementing motion-driven virtual avatar interaction mainly suffer from the following limitations:

[0004] First, it is highly dependent on hardware. Most traditional motion capture systems require specialized equipment, such as wearable inertial sensors or depth cameras, which results in high hardware costs, limited deployment scenarios, and is not suitable for ordinary users.

[0005] Secondly, it has poor adaptability. Most current 3D interactive systems are developed based on native applications and cannot run directly across platforms, especially in the more convenient and faster browser environment, which limits their popularity on multiple terminal devices.

[0006] Secondly, the interaction chain is incomplete. Most existing pure camera-driven solutions are limited to the level of "face-driven" or "upper body-driven", lacking a complete chain that can simultaneously capture full-body posture, fine hand movements and combine them with physical interaction.

[0007] Furthermore, the interaction logic of virtual items is simple. In existing technologies, the interaction between virtual items and virtual avatars is mostly static attachment, such as wearing them by clicking. It cannot respond to the user's dynamic actions, such as turning around to view the details of the item or reaching out to grab the virtual item, and lacks a natural interactive experience. Summary of the Invention

[0008] This application provides an interactive method, system, and electronic device based on a web front-end, to at least solve the problem of poor adaptability of virtual avatar interaction methods in related technologies.

[0009] In a first aspect, embodiments of this application provide an interaction method based on a Web front-end, the interaction method comprising:

[0010] The browser accesses the terminal camera to obtain the user's video stream, and processes the video stream in the browser front end to extract key points of the human body.

[0011] The human body key points are mapped to the skeletal animation parameters of the virtual image through a layered motion mapping engine.

[0012] Based on the preset attributes of the virtual item and the skeletal animation parameters, the relative pose of the virtual item and the virtual image is calculated;

[0013] In a browser environment, the interactive screen of the virtual item and the virtual avatar is rendered and displayed based on the relative pose.

[0014] In some embodiments, mapping the human keypoints to the skeletal animation parameters of the virtual avatar using a hierarchical motion mapping engine includes:

[0015] At the underlying data processing level, the Kalman filter algorithm is used to smooth the key point data of the human body;

[0016] In the middle layer of the algorithm, the smoothed human keypoint data is converted into rotation angles of the corresponding skeletal nodes of the virtual image using an inverse dynamics algorithm, thereby obtaining the skeletal animation parameters, wherein:

[0017] At the logical level, multiple action states are preset through a state machine, and state switching instructions are output based on the skeletal animation parameters and preset judgment thresholds.

[0018] In some embodiments, the action states include: a stationary state, a turning state, and a grabbing state. Based on the skeletal animation parameters and a preset threshold, the output state switching command includes:

[0019] Based on the key points of the human body, the rotation angle of the user's spinal bones is obtained. When the rotation angle is greater than a preset angle threshold, a state switching command is output to switch from the static state to the turning state.

[0020] Based on the key points of the human body, calculate the real-time spatial distance between the user's hand and the virtual object. When the spatial distance is less than a preset distance threshold, output a state switching command from the current state to the grasping state.

[0021] Each of the state switching instructions is configured with priority information. When multiple state switching trigger conditions are simultaneously met, the state switching instruction is output according to the priority information.

[0022] In some embodiments, processing the video stream in the browser front-end to extract human key points includes:

[0023] Under HTTPS, the video stream is obtained through the browser's application programming interface;

[0024] The image frames in the video stream are forwarded to the pose detection model, hand key point detection model, and face key point detection model deployed in the browser environment.

[0025] By executing the pose detection model, the hand keypoint detection model, and the face keypoint detection model in the Web Worker thread in the browser background, the image frame is processed concurrently, and the pose keypoints, hand keypoints, and face keypoints in the image frame are extracted respectively to obtain the human body keypoints.

[0026] In some embodiments, calculating the relative pose of the virtual item and the virtual avatar includes:

[0027] By parsing the JSON configuration file corresponding to the virtual item, the preset attributes of the virtual item are obtained, wherein the preset attributes include: item type, attachment point and physical characteristics;

[0028] Based on the skeletal animation parameters, the rotation angle of the spine of the virtual character is obtained, and based on the rotation angle and the preset attributes, the relative following pose of the virtual item with respect to the pose key points is calculated.

[0029] If the collision radius of the user's hand keypoint overlaps with that of the virtual object, the attachment relationship of the virtual object is switched to the hand keypoint. Based on the skeletal animation parameters and the preset attributes, the relative following pose of the virtual object relative to the hand keypoint is calculated.

[0030] In some embodiments, the method further includes:

[0031] In the JSON configuration file corresponding to the virtual item, the variable presentation state of the virtual item and its corresponding triggering conditions are defined according to the physical characteristics;

[0032] Determine whether the interactive action generated by the human body key points meets the triggering condition;

[0033] If so, based on the triggering condition, update the rendering properties of the 3D model of the virtual item, and / or trigger the animation effect corresponding to the 3D model.

[0034] In some embodiments, the method further includes:

[0035] The JSON configuration file is used to configure the triggering conditions for the linkage between the first virtual item and the second virtual item;

[0036] Detect whether the collision between the first virtual item and the second virtual item satisfies the linkage triggering condition;

[0037] If so, update the rendering properties of the 3D models of the first virtual item and the second virtual item, and / or trigger their corresponding associated animation effects.

[0038] In some embodiments, the method further includes:

[0039] The Draco compression algorithm is used to compress the geometric data of the three-dimensional models of the virtual avatar and the virtual item, wherein the three-dimensional models of the virtual avatar and the virtual item are stored in Gltf format;

[0040] The 3D models and texture resources of the virtual avatars and virtual items are cached through Service Worker;

[0041] In a browser environment, based on the relative poses of the virtual item and the virtual image, as well as the 3D models and texture resources of the virtual image and the virtual item loaded offline from ServiceWorker, the interactive screen of the virtual item and the virtual image is rendered and displayed.

[0042] Secondly, embodiments of this application provide an interactive system based on a web front-end, the interactive system comprising: a data acquisition module, an action processing module, and a rendering module, wherein:

[0043] The acquisition module is used to access the user's video stream through the browser's terminal camera and process the video stream in the browser front end to extract key points of the human body.

[0044] The motion processing module is used to map the human body key points to the skeletal animation parameters of the virtual image through a hierarchical motion mapping engine.

[0045] Furthermore, based on the preset attributes of the virtual item and the skeletal animation parameters, the relative pose of the virtual item and the virtual image is calculated;

[0046] The rendering module is used to render and display the interactive screen of the virtual item and the virtual image in a browser environment based on the relative pose.

[0047] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0049] Compared to related technologies, this application provides a web-based front-end interaction method and system. By extracting key points of the user's limbs, face, and hands using a regular camera in the browser front-end, and constructing a hierarchical motion mapping engine, combined with Kalman filtering, inverse kinematics (IK) algorithms, and state machines, it achieves low-latency, highly natural virtual avatar motion mapping. Simultaneously, by establishing a standardized virtual item interaction framework, based on JSON configuration attributes and dynamic attachment logic, it enables complex physical interactions between virtual items and virtual avatars based on the user's real actions. This solution solves the problems of high hardware dependence, poor cross-platform compatibility, simple interaction links, and severe latency and distortion inherent in existing technologies. It requires no dedicated hardware, lowering the user threshold, and achieves cross-platform compatibility based on the web technology stack. Through lightweight front-end algorithms and rendering optimization, it ensures smooth operation even on low-configuration devices, significantly improving the real-time performance, immersion, and realism of the interaction, providing efficient, convenient, and expressive technical support for application scenarios such as virtual try-on and online social networking. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a schematic diagram illustrating an application scenario of the web-based front-end interaction method provided in the embodiments of this application;

[0052] Figure 2 This is a flowchart of an interaction method based on a web front-end according to an embodiment of this application;

[0053] Figure 3 This is a structural block diagram of a web-based interactive system according to one embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0056] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0057] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0058] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0059] In this document, it should be understood that the terms used may be technical means used to implement part of this invention or other summary technical terms. For example, the terms may include:

[0060] WebGL (Web Graphics Library) refers to a JavaScript-based 3D graphics standard used to implement hardware-accelerated 3D rendering in browsers.

[0061] API (Application Programming Interface): refers to the interaction interface between software components, such as the interface used to call the camera and process visual data in this article.

[0062] Virtual Character: A digital avatar built based on a 3D model, which can simulate the posture and expression of a real user through motion mapping.

[0063] Motion capture refers to the process of extracting user's limb, facial, or hand movement data using computer vision technology. In this article, it specifically refers to non-invasive motion capture based on monocular or binocular cameras.

[0064] Attitude estimation refers to the technique of using algorithms to predict the position and angle of key human body nodes (such as joints) from images and generate skeletal structure data.

[0065] Virtual Goods: refers to digital 3D items (such as backpacks, watches, hats, etc.) used to interact with virtual avatars, and have collision detection and attachment logic with virtual avatars.

[0066] Mapping Algorithm: A method for calculating parameters that convert real user motion data into skeletal animation parameters for a virtual avatar, ensuring that the virtual avatar's movements are consistent with the user's movements.

[0067] Gltf (GL Transmission Format): refers to an efficient 3D model format suitable for model loading and rendering in a web environment.

[0068] This application provides a web-based interactive method applicable to scenarios such as virtual try-on, online social networking, and digital twin displays. This method aims to achieve motion capture and real-time virtual interaction with relatively low-threshold hardware requirements.

[0069] Figure 1 This is a schematic diagram illustrating an application scenario of the web-based front-end interaction method provided in the embodiments of this application, such as... Figure 1As shown, the solution in this application can be applied to an interactive scenario entirely based on a web browser environment. In this application environment, the user uses a terminal device with a built-in camera, accesses a service page through a browser, and grants camera permissions. The terminal device captures a video stream containing the user in real time and processes the video stream data in the browser frontend.

[0070] The terminal device generates a basic virtual avatar based on the user's image using a Gaussian virtual human algorithm. Simultaneously, a computer vision model deployed at the front end captures the user's limb, facial, and hand movements from the video stream in real time, extracting 3D keypoint coordinates. This coordinate data is then processed through smoothing filtering and inverse kinematics (IK) algorithms to convert it into skeletal animation parameters for the virtual avatar, enabling it to perform natural, synchronized movements with the user in real time. When the user interacts with virtual objects (such as backpacks or hats), the relative position and posture between the virtual object and the virtual avatar are dynamically calculated and adjusted based on the user's specific actions (such as turning or raising a hand to grab). Finally, the driven virtual avatar and the dynamically interacting virtual objects are synthesized in real time, and the final interactive scene is visualized on the user's terminal screen. The entire process has extremely low latency, ensuring a smooth and immersive experience.

[0071] exist Figure 1 In the application scenarios shown, the terminal device can be any device with a regular camera and browser, such as a personal computer, smartphone, tablet, or VR all-in-one device, without the need to install additional applications or equip it with dedicated sensors.

[0072] Figure 2 This is a flowchart illustrating a web front-end-based interactive method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0073] S201 uses the browser to access the terminal camera, acquires the user's video stream, and processes the video stream in the browser front-end to extract key points of the human body.

[0074] In this embodiment, this step aims to achieve non-intrusive capture of user motion data by using a standard web technology and the user terminal's built-in ordinary camera as the sole input device.

[0075] Specifically, firstly, under an HTTPS page that conforms to security protocols, the browser's application programming interface (API), such as the get User Media API, is used to call the front or rear camera of the terminal device (such as a personal computer, smartphone, or tablet) to obtain the user's real-time video stream.

[0076] Optionally, to balance clarity and processing efficiency, the video stream resolution can be set to 1280x720 and the frame rate to 30fps.

[0077] Furthermore, after acquiring the video stream, in order to achieve cross-platform operation and avoid delays caused by complex backend data transmission and processing, this embodiment deploys the visual recognition and key point extraction model in the browser front-end environment for execution.

[0078] Specifically, consecutive image frames from the video stream are concurrently forwarded to multiple visual detection models deployed in the browser environment, including pose detection models (such as Media Pipe Pose), hand keypoint detection models (such as MediaPipe Hands), and face keypoint detection models (called via Face-API).

[0079] Additionally, it should be noted that in this embodiment, to ensure the smoothness of the UI and avoid visual computation blocking the main thread, the model's computational tasks are executed in a Web Worker thread in the browser's background. Each model processes image frames concurrently, extracting pose keypoints (e.g., 33 limb key nodes), hand keypoints (e.g., 21 hand joint nodes), and facial keypoints (e.g., 68 facial feature points) from the images, collectively forming a complete set of human keypoint data. The extracted keypoint coordinates are initially in an image-based coordinate system, which needs to be further transformed to a world coordinate system centered on the user's hip, providing a unified three-dimensional spatial reference for subsequent motion mapping.

[0080] Step S201 uses a browser front-end visual model to capture key points of the whole body, hands, and face in real time using a regular camera, and leverages Web Worker to ensure smooth processing. This achieves low-hardware-cost, cross-platform, and refined motion data acquisition, laying the foundation for subsequent natural interaction.

[0081] S202 uses a layered motion mapping engine to map human body key points to the skeletal animation parameters of the virtual avatar.

[0082] This step aims to convert the potentially noisy discrete keypoint data extracted from the front end into smooth, natural virtual avatar skeletal animation parameters that conform to human kinematics. In this embodiment, a layered motion mapping engine is used to implement this conversion process.

[0083] Specifically, at the data processing level, in order to solve the problem of key point data jitter caused by changes in light, occlusion or rapid movement during camera capture, a Kalman filter algorithm is used to smooth the original human key point data.

[0084] The following iterative process of prediction and update is used to reduce data noise and control the overall action delay within 100ms: Predicted value = Actual value of the previous frame + speed × time difference Actual value = Predicted value + gain × (current detection value - predicted value);

[0085] In the middle layer of the algorithm, the smoothed human keypoint data is converted into the rotation angle of the corresponding skeletal nodes of the virtual image through the inverse kinematics (IK) algorithm, thus obtaining the skeletal animation parameters.

[0086] For example, by taking the three-dimensional keypoint positions of the user's shoulder, elbow, and wrist as input, the IK algorithm can calculate the angles that the shoulder, elbow, and wrist joints in the virtual avatar's arm need to rotate, so that the end of the virtual avatar's hand can accurately reach the spatial position corresponding to the user's hand, ensuring the naturalness and physical realism of the virtual avatar's movements.

[0087] At the logical level, a state machine manages the macroscopic action states of the virtual avatar, with multiple preset action states, such as "still state," "turning state," and "grabbing state." This state machine outputs state switching instructions based on skeletal animation parameters and preset judgment thresholds.

[0088] For example, the system continuously monitors the user's spinal rotation angle calculated from key points on the human body. When this rotation angle exceeds a preset threshold (e.g., 30°), the state machine outputs a command to switch from a "stationary state" to a "turning state." Simultaneously, it calculates the 3D spatial distance between the user's hand key points and virtual objects in the scene in real time. When this distance is less than a preset distance threshold (e.g., 30 pixels), it triggers and outputs a command to switch to a "grabbing state." To handle complex actions, each state switching command can be configured with priority information. When multiple state switching trigger conditions are met simultaneously, only the command with the highest priority is output based on the priority information, avoiding action conflicts.

[0089] Step S202 transforms the original key points into smooth, natural, and logically clear virtual image animation parameters through filtering, inverse dynamics, and state machine layered processing. These parameters are used for the interaction of virtual objects in subsequent interactions, achieving low-latency and high-fidelity motion mapping and significantly improving the realism of the virtual image.

[0090] S203, calculate the relative pose of the virtual item and the virtual image based on the preset attributes and skeletal animation parameters of the virtual item.

[0091] To achieve natural and scalable interaction between virtual items and virtual avatars, this embodiment designs an adaptive virtual item interaction framework. First, by parsing the JSON configuration file corresponding to each virtual item, its preset attributes are obtained. These attributes define key information about the item, such as item type (e.g., wearable item), attachment point (e.g., back, shoulder, wrist), and physical characteristics (e.g., collision radius).

[0092] Specifically, during pose calculation, the real-time rotation angle of a specific bone (such as the spine) of the virtual character is obtained based on the skeletal animation parameters output by S102. Based on this rotation angle and the attachment point information defined in the item configuration file, the relative following pose of the virtual item with respect to the pose key points can be calculated.

[0093] For example, when the attachment point of a "shoulder bag" is defined as "left shoulder", even if the user's virtual avatar turns around, the shoulder bag model will adjust synchronously according to the pose change of the virtual avatar's shoulder bones to maintain the visual effect of it always hanging on the shoulder.

[0094] Furthermore, to achieve dynamic interaction, the technical solution provided in this embodiment continuously detects whether the user's hand keypoints overlap with the collision radius of the virtual object. When overlap is detected, such as when the user reaches out to touch a backpack, the attachment relationship of the virtual object is dynamically switched to the user's hand keypoints. Subsequently, based on the skeletal animation parameters of the hand and the preset attributes of the object, the relative following pose of the virtual object relative to the hand keypoints is recalculated to achieve virtual grasping and movement effects.

[0095] Step S203 simplifies the adaptation process for new items through standardized JSON configuration and a dynamic attachment algorithm, solving the problems of developing separate logic for different items and high adaptation costs in existing solutions. Simultaneously, by dynamically calculating the item's following pose and switching attachment relationships based on collision detection, it enables natural interaction between virtual items and virtual avatars based on the user's real actions, supporting complex logic such as turning and following, and dynamic grabbing, greatly enhancing the immersiveness, realism, and scalability of the interaction.

[0096] S204, in a browser environment, renders and displays interactive screens of virtual items and virtual avatars based on their relative poses.

[0097] The purpose of this step is to efficiently present the dynamic relationship between the aforementioned calculated virtual images and virtual items on the user's browser interface, ensuring the real-time performance and smoothness of the interactive screen. In this embodiment, optionally, the rendering system is built based on the Three.js and WebGL technology stack.

[0098] It should also be noted that several optimization measures were taken to improve performance and loading speed. First, the 3D models were lightweighted, using the Gltf format to store the models of virtual avatars and items, and the Draco compression algorithm was used to compress the geometric data of the models at a high rate, reducing the model file size by more than 50%. Second, to accelerate secondary loading and support offline access, the 3D models and texture resources of virtual avatars and items were cached using Service Worker technology on the browser side.

[0099] During the rendering loop, the rendering module synthesizes the final interactive screen in real time based on the latest relative poses of the virtual image and virtual objects calculated by S103, as well as the 3D models and texture resources loaded from Service Worker or the network, and ensures that the rendering frame rate is not less than 30fps, providing users with smooth, low-latency visual feedback.

[0100] Step S204 utilizes front-end optimization technologies such as model compression, hierarchical rendering, and resource caching to achieve high-performance real-time 3D rendering in a standard browser environment, ensuring a smooth and high-quality virtual interactive visual experience even on low-configuration devices.

[0101] Furthermore, to further enrich the details and realism of the interaction, the method in this embodiment also includes: pre-defining one or more variable presentation states of the virtual item in the JSON configuration file corresponding to the virtual item, based on its physical characteristics, and the state switching trigger conditions corresponding to each state. During operation, it is determined in real time whether the interactive actions generated by the user's key points satisfy the aforementioned preset state trigger conditions. If it is determined that they are satisfied, the rendering attributes of the virtual item's 3D model will be updated accordingly based on the trigger conditions, and / or specific animation effects bound to it will be triggered.

[0102] For example, for a virtual backpack item, an "open" state can be defined in its JSON file, triggered by "the collision radius of the user's other hand's key point overlapping with the backpack's buckle model." When this "open" action is detected in real time, the backpack model's preset "open" animation is triggered, and the rendering properties of its internal materials are updated from invisible to visible, thus simulating a realistic backpack opening effect.

[0103] Furthermore, to achieve more complex inter-item interaction (i.e., one item affects another item), the method in this embodiment also includes configuring linkage triggering conditions for the first virtual item (such as "key") and the second virtual item (such as "treasure chest") in a JSON configuration file.

[0104] For example, after a user picks up a "key" in a "grab" state, the key's collider, driven by the user's hand's key points, is moved near the lock cylinder collider of the "treasure chest." When the system detects in real time that the two meet preset linkage trigger conditions (e.g., colliders overlap and the key model's rotation angle meets a specific threshold), a related animation effect will be triggered.

[0105] First virtual item (key): Play the "rotate insertion" animation and update its attachment point so that it "attaches" to the lock cylinder;

[0106] Second virtual item (treasure chest): Its lock cylinder model's rendering attribute is updated to "invisible", and at the same time, its "lid opens" animation is triggered, revealing the internal model.

[0107] This configuration enables realistic interaction between virtual items, greatly enriching the depth and fun of the interaction.

[0108] Through steps S101 to S103 above, this solution solves the problems of high hardware dependence, poor cross-platform compatibility, simple interaction links, and severe latency and distortion in existing technologies. It eliminates the need for dedicated hardware, lowering the barrier to entry for users, and achieves cross-platform compatibility based on the Web technology stack. Through lightweight front-end algorithms and rendering optimization, it ensures smooth operation even on low-configuration devices, significantly improving the real-time performance, immersion, and realism of the interaction, providing efficient, convenient, and expressive technical support for application scenarios such as virtual try-on and online social networking.

[0109] On the other hand, embodiments of this application also provide an interactive system based on a web front-end, which is used to implement the above-described method. Figure 3 This is a structural block diagram of a web-based interactive system according to one embodiment of the present application, such as... Figure 3 As shown, the system includes: a data acquisition module 30, an action processing module 31, and a rendering module 32.

[0110] The acquisition module 30 is used to access the terminal camera through the browser, acquire the user's video stream, and process the video stream in the browser front end to extract human body key points.

[0111] In this embodiment, this step aims to achieve non-intrusive capture of user motion data by using a standard web technology and the user terminal's built-in ordinary camera as the sole input device.

[0112] Specifically, firstly, under an HTTPS page that conforms to security protocols, the browser's application programming interface (API), such as the get User Media API, is used to call the front or rear camera of the terminal device (such as a personal computer, smartphone, or tablet) to obtain the user's real-time video stream. Optionally, to balance clarity and processing efficiency, the video stream resolution is set to 1280x720 and the frame rate is set to 30fps.

[0113] Furthermore, after acquiring the video stream, in order to achieve cross-platform operation and avoid delays caused by complex backend data transmission and processing, this embodiment deploys the visual recognition and key point extraction model in the browser front-end environment for execution.

[0114] The acquisition module 30 uses a browser front-end visual model to capture key points of the whole body, hands, and face in real time using a regular camera, and leverages Web Worker to ensure smooth processing. This achieves low-hardware-cost, cross-platform, and refined motion data acquisition, providing a foundation for subsequent natural interaction.

[0115] The motion processing module 31 is used to map human key points to the skeletal animation parameters of the virtual image through a layered motion mapping engine; and to calculate the relative pose of the virtual object and the virtual image based on the preset attributes and skeletal animation parameters of the virtual object.

[0116] This step aims to convert the potentially noisy discrete keypoint data extracted from the front end into smooth, natural virtual avatar skeletal animation parameters that conform to human kinematics. In this embodiment, a layered motion mapping engine is used to implement this conversion process.

[0117] Specifically, at the data processing level, in order to solve the problem of key point data jitter caused by changes in light, occlusion or rapid movement during camera capture, a Kalman filter algorithm is used to smooth the original human key point data.

[0118] In the middle layer of the algorithm, the smoothed human keypoint data is converted into the rotation angle of the corresponding skeletal nodes of the virtual image through the inverse kinematics (IK) algorithm, thus obtaining the skeletal animation parameters.

[0119] At the logical level, a state machine manages the macroscopic action states of the virtual avatar, with multiple preset action states, such as "still state," "turning state," and "grabbing state." This state machine outputs state switching instructions based on skeletal animation parameters and preset judgment thresholds.

[0120] Furthermore, to achieve natural and scalable interaction between virtual items and virtual avatars, this embodiment designs an adaptive virtual item interaction framework. First, by parsing the JSON configuration file corresponding to each virtual item, its preset attributes are obtained. These attributes define key information about the item, such as item type (e.g., wearable item), attachment point (e.g., back, shoulder, wrist), and physical characteristics (e.g., collision radius).

[0121] During pose calculation, skeletal animation parameters are used to obtain the real-time rotation angles of specific bones (such as the spine) of the virtual character. Based on these rotation angles and the attachment point information defined in the item configuration file, the relative following pose of the virtual item with respect to the pose keypoints can be calculated.

[0122] The motion processing module 31 transforms the original key points into smooth, natural, and logically clear virtual image animation parameters through filtering, inverse dynamics, and state machine layered processing, which are used for the interaction of virtual objects in subsequent interactions. Furthermore, through standardized JSON configuration and dynamic attachment algorithm, the adaptation process of new items is simplified. By dynamically calculating the item's following pose and switching the attachment relationship based on collision detection, natural interaction between virtual items and virtual images based on the user's real actions is realized, improving the immersion, realism, and scalability of the interaction.

[0123] The rendering module 32 is used to render and display interactive screens of virtual items and virtual images in a browser environment based on their relative poses.

[0124] The purpose of the rendering module 32 is to efficiently present the dynamic relationship between the aforementioned calculated virtual images and virtual items on the user's browser interface, ensuring the real-time performance and smoothness of the interactive screen. In this embodiment, the rendering system is built on the Three.js and WebGL technology stack. By comprehensively utilizing front-end optimization technologies such as model compression, hierarchical rendering, and resource caching, high-performance real-time 3D rendering is achieved in a standard browser environment, ensuring a smooth and high-quality virtual interactive visual experience even on low-configuration devices.

[0125] This system addresses the problems of high hardware dependence, poor cross-platform compatibility, simple interaction links, and severe latency and distortion inherent in existing technologies. It eliminates the need for dedicated hardware, lowering the barrier to entry, and achieves cross-platform compatibility based on the Web technology stack. Through lightweight front-end algorithms and rendering optimizations, it ensures smooth operation even on low-configuration devices, significantly improving the real-time performance, immersion, and realism of the interaction. This provides efficient, convenient, and expressive technical support for applications such as virtual try-on and online social networking.

[0126] In one embodiment, Figure 4This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operating system to run, the computer programs are executed by the processor to implement a web-based front-end interaction method, and the database stores data.

[0127] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to computer program instructions. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0129] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An interactive method based on a Web front-end, characterized in that, The interaction method includes: The browser accesses the terminal camera to obtain the user's video stream, and processes the video stream in the browser front end to extract human key points. The human body key points are mapped to the skeletal animation parameters of the virtual image through a layered motion mapping engine. Based on the preset attributes of the virtual item and the skeletal animation parameters, the relative pose of the virtual item and the virtual image is calculated, specifically including: obtaining the preset attributes of the virtual item by parsing the JSON configuration file corresponding to the virtual item, wherein the preset attributes include: item type, attachment point and physical characteristics; Based on the skeletal animation parameters, the rotation angle of the spine of the virtual character is obtained, and based on the rotation angle and the preset attributes, the relative following pose of the virtual item with respect to the pose key points is calculated. Detect whether the user's hand keypoint overlaps with the collision radius of the virtual object. If so, switch the attachment relationship of the virtual object to the hand keypoint, and calculate the relative following pose of the virtual object relative to the hand keypoint based on the skeletal animation parameters and the preset attributes. In a browser environment, the interactive screen of the virtual item and the virtual image is rendered and displayed according to the relative pose; The method further includes: Configure the linkage triggering conditions for the first and second virtual items in the JSON configuration file corresponding to the virtual item; Detect whether the collision between the first virtual item and the second virtual item satisfies the linkage triggering condition; If so, update the rendering properties of the 3D models of the first virtual item and the second virtual item, and / or trigger their corresponding associated animation effects.

2. The interaction method according to claim 1, characterized in that, The layered motion mapping engine maps the human body key points to the skeletal animation parameters of the virtual avatar, including: At the underlying data processing level, the Kalman filter algorithm is used to smooth the key point data of the human body; In the middle layer of the algorithm, the smoothed human keypoint data is converted into rotation angles of the corresponding skeletal nodes of the virtual image using an inverse dynamics algorithm, thereby obtaining the skeletal animation parameters, wherein: At the logical level, multiple action states are preset through a state machine, and state switching instructions are output based on the skeletal animation parameters and preset judgment thresholds.

3. The interaction method according to claim 2, characterized in that, The action states include: stationary state, turning state, and grabbing state. Based on the skeletal animation parameters and a preset threshold, the output state switching instructions include: Based on the key points of the human body, the rotation angle of the user's spinal bones is obtained. When the rotation angle is greater than a preset angle threshold, a state switching command is output to switch from the static state to the turning state. Based on the key points of the human body, calculate the real-time spatial distance between the user's hand and the virtual object. When the spatial distance is less than a preset distance threshold, output a state switching command from the current state to the grasping state. Each of the state switching instructions is configured with priority information. When multiple state switching trigger conditions are simultaneously met, the state switching instruction is output according to the priority information.

4. The interaction method according to claim 1, characterized in that, Processing the video stream in the browser front-end to extract human key points includes: Under HTTPS, the video stream is obtained through the browser's application programming interface; The image frames in the video stream are forwarded to the pose detection model, hand key point detection model, and face key point detection model deployed in the browser environment. By executing the pose detection model, the hand keypoint detection model, and the face keypoint detection model in the Web Worker thread in the browser background, the image frame is processed concurrently, and the pose keypoints, hand keypoints, and face keypoints in the image frame are extracted respectively to obtain the human body keypoints.

5. The interaction method according to claim 1, characterized in that, The method further includes: In the JSON configuration file corresponding to the virtual item, the variable presentation state of the virtual item and its corresponding triggering conditions are defined according to the physical characteristics; Determine whether the interactive action generated by the human body key points meets the triggering condition; If so, based on the triggering condition, update the rendering properties of the 3D model of the virtual item, and / or trigger the animation effect corresponding to the 3D model.

6. The interaction method according to claim 1, characterized in that, The method further includes: The Draco compression algorithm is used to compress the geometric data of the three-dimensional models of the virtual avatar and the virtual item, wherein the three-dimensional models of the virtual avatar and the virtual item are stored in Gltf format; The 3D models and texture resources of the virtual avatars and virtual items are cached through Service Worker; In a browser environment, based on the relative poses of the virtual item and the virtual image, as well as the 3D models and texture resources of the virtual image and the virtual item loaded offline from ServiceWorker, the interactive screen of the virtual item and the virtual image is rendered and displayed.

7. An interactive system based on a web front-end, characterized in that, The interactive system includes: a data acquisition module, an action processing module, and a rendering module, wherein: The acquisition module is used to access the user's video stream through the browser's terminal camera and process the video stream in the browser front end to extract key points of the human body. The motion processing module is used to map the human body key points to the skeletal animation parameters of the virtual image through a hierarchical motion mapping engine. Furthermore, based on the preset attributes of the virtual item and the skeletal animation parameters, the relative pose of the virtual item and the virtual image is calculated, specifically including: obtaining the preset attributes of the virtual item by parsing the JSON configuration file corresponding to the virtual item, wherein the preset attributes include: item type, attachment point and physical characteristics; Based on the skeletal animation parameters, the rotation angle of the spine of the virtual character is obtained, and based on the rotation angle and the preset attributes, the relative following pose of the virtual item with respect to the pose key points is calculated. Detect whether the user's hand keypoint overlaps with the collision radius of the virtual object. If so, switch the attachment relationship of the virtual object to the hand keypoint, and calculate the relative following pose of the virtual object relative to the hand keypoint based on the skeletal animation parameters and the preset attributes. The rendering module is used to render and display the interactive screen of the virtual item and the virtual image in a browser environment according to the relative pose, wherein the linkage triggering conditions of the first virtual item and the second virtual item are configured in the JSON configuration file corresponding to the virtual item. Detect whether the collision between the first virtual item and the second virtual item satisfies the linkage triggering condition; If so, update the rendering properties of the 3D models of the first virtual item and the second virtual item, and / or trigger their corresponding associated animation effects.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

Images