Interaction method and system based on Web front end and electronic equipment

Abstract

The invention relates to an interaction method and system based on a Web front end and electronic equipment, and the method comprises the steps: calling a terminal camera through a browser, obtaining a video stream of a user, and processing the video stream at the front end of the browser to extract human body key points; mapping the human body key points into skeleton animation parameters of the virtual image through a hierarchical action mapping engine; calculating a relative pose of the virtual article and the virtual image according to a preset attribute and a skeleton animation parameter of the virtual article; and in a browser environment, according to the relative pose, rendering and displaying an interaction picture of the virtual article and the virtual image. The problems of high hardware dependency, poor cross-platform compatibility, simple interaction link and serious delay and distortion in the prior art are solved. Special hardware is not needed, the use threshold of a user is lowered, and cross-platform compatibility is achieved based on a Web technology stack.

Description

Technical Field

[0001] This application relates to the field of virtual reality, and in particular to an interactive method, system, and electronic device based on a web front-end. Background Technology

[0002] With the development of internet technology, users' demands for immersive and realistic online interactions are increasing. Traditional virtual avatar interactions often rely on preset animations or simple clicks, failing to respond to users' natural movements in real time, resulting in a stiff and unrealistic interactive experience.

[0003] In existing technologies, methods for implementing motion-driven virtual avatar interaction mainly suffer from the following limitations: First, it is highly dependent on hardware. Most traditional motion capture systems require specialized equipment, such as wearable inertial sensors or depth cameras, which results in high hardware costs, limited deployment scenarios, and is not suitable for ordinary users.

[0004] Secondly, it has poor adaptability. Most current 3D interactive systems are developed based on native applications and cannot run directly across platforms, especially in the more convenient and faster browser environment, which limits their popularity on multiple terminal devices.

[0005] Secondly, the interaction chain is incomplete. Most existing pure camera-driven solutions are limited to the level of "face-driven" or "upper body-driven", lacking a complete chain that can simultaneously capture full-body posture, fine hand movements and combine them with physical interaction.

[0006] Furthermore, the interaction logic of virtual items is simple. In existing technologies, the interaction between virtual items and virtual avatars is mostly static attachment, such as wearing them by clicking. It cannot respond to the user's dynamic actions, such as turning around to view the details of the item or reaching out to grab the virtual item, and lacks a natural interactive experience. Summary of the Invention

[0007] This application provides an interactive method, system, and electronic device based on a web front-end, to at least solve the problem of poor adaptability of virtual avatar interaction methods in related technologies.

[0008] In a first aspect, embodiments of this application provide an interaction method based on a Web front-end, the interaction method comprising: The browser accesses the terminal camera to obtain the user's video stream, and processes the video stream in the browser front end to extract human key points. The human body key points are mapped to the skeletal animation parameters of the virtual image through a layered motion mapping engine. Based on the preset attributes of the virtual item and the skeletal animation parameters, the relative pose of the virtual item and the virtual image is calculated; In a browser environment, the interactive screen of the virtual item and the virtual image is rendered and displayed based on the relative pose.

[0009] In some embodiments, mapping the human keypoints to skeletal animation parameters of the virtual avatar using a hierarchical motion mapping engine includes: At the underlying data processing level, the Kalman filter algorithm is used to smooth the key point data of the human body; In the middle layer of the algorithm, the smoothed human keypoint data is converted into rotation angles of the corresponding skeletal nodes of the virtual image using an inverse dynamics algorithm, thereby obtaining the skeletal animation parameters, wherein: At the logical level, multiple action states are preset through a state machine, and state switching instructions are output based on the skeletal animation parameters and preset judgment thresholds.

[0010] In some embodiments, the action states include: a stationary state, a turning state, and a grasping state. Based on the skeletal animation parameters and a preset threshold, the output state switching command includes: Based on the key points of the human body, the rotation angle of the user's spinal bones is obtained. When the rotation angle is greater than a preset angle threshold, a state switching command is output to switch from the static state to the turning state. Based on the key points of the human body, calculate the real-time spatial distance between the user's hand and the virtual object. When the spatial distance is less than a preset distance threshold, output a state switching command from the current state to the grasping state. Each of the state switching instructions is configured with priority information. When multiple state switching trigger conditions are simultaneously met, the state switching instruction is output according to the priority information.

[0011] In some embodiments, processing the video stream in the browser front-end to extract human key points includes: Under HTTPS, the video stream is obtained through the browser's application programming interface; The image frames in the video stream are forwarded to the pose detection model, hand key point detection model, and face key point detection model deployed in the browser environment. By executing the pose detection model, the hand keypoint detection model, and the face keypoint detection model in the Web Worker thread in the browser background, the image frame is processed concurrently, and the pose keypoints, hand keypoints, and face keypoints in the image frame are extracted respectively to obtain the human body keypoints.

[0012] In some embodiments, calculating the relative pose of the virtual item and the virtual avatar includes: By parsing the JSON configuration file corresponding to the virtual item, the preset attributes of the virtual item are obtained, wherein the preset attributes include: item type, attachment point and physical characteristics; Based on the skeletal animation parameters, the rotation angle of the spine of the virtual character is obtained, and based on the rotation angle and the preset attributes, the relative following pose of the virtual item with respect to the pose key points is calculated. If the collision radius of the user's hand keypoint overlaps with that of the virtual object, the attachment relationship of the virtual object is switched to the hand keypoint. Based on the skeletal animation parameters and the preset attributes, the relative following pose of the virtual object relative to the hand keypoint is calculated.

[0013] In some embodiments, the method further includes: In the JSON configuration file corresponding to the virtual item, the variable presentation state of the virtual item and its corresponding triggering conditions are defined according to the physical characteristics; Determine whether the interactive action generated by the human body key points meets the triggering condition; If so, based on the triggering condition, update the rendering properties of the 3D model of the virtual item, and / or trigger the animation effect corresponding to the 3D model.

[0014] In some embodiments, the method further includes: The JSON configuration file is used to configure the triggering conditions for the linkage between the first virtual item and the second virtual item; Detect whether the collision between the first virtual item and the second virtual item satisfies the linkage triggering condition; If so, update the rendering properties of the 3D models of the first virtual item and the second virtual item, and / or trigger their corresponding associated animation effects.

[0015] In some embodiments, the method further includes: The Draco compression algorithm is used to compress the geometric data of the three-dimensional models of the virtual avatar and the virtual item, wherein the three-dimensional models of the virtual avatar and the virtual item are stored in Gltf format; The 3D models and texture resources of the virtual avatars and virtual items are cached through Service Worker; In a browser environment, based on the relative poses of the virtual item and the virtual image, as well as the 3D models and texture resources of the virtual image and the virtual item loaded offline from ServiceWorker, the interactive screen of the virtual item and the virtual image is rendered and displayed.

[0016] Secondly, embodiments of this application provide an interactive system based on a web front-end, the interactive system comprising: a data acquisition module, an action processing module, and a rendering module, wherein: The acquisition module is used to access the user's video stream through the browser's terminal camera and process the video stream in the browser front end to extract key points of the human body. The motion processing module is used to map the human body key points to the skeletal animation parameters of the virtual image through a hierarchical motion mapping engine. Furthermore, based on the preset attributes of the virtual item and the skeletal animation parameters, the relative pose of the virtual item and the virtual image is calculated; The rendering module is used to render and display the interactive screen of the virtual item and the virtual image in a browser environment based on the relative pose.

[0017] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0019] Compared to related technologies, this application provides a web-based front-end interaction method and system. By extracting key points of the user's limbs, face, and hands using a regular camera in the browser front-end, and constructing a hierarchical motion mapping engine, combined with Kalman filtering, inverse kinematics (IK) algorithms, and state machines, it achieves low-latency, highly natural virtual avatar motion mapping. Simultaneously, by establishing a standardized virtual item interaction framework, based on JSON configuration attributes and dynamic attachment logic, it enables complex physical interactions between virtual items and virtual avatars based on the user's real actions. This solution solves the problems of high hardware dependence, poor cross-platform compatibility, simple interaction links, and severe latency and distortion inherent in existing technologies. It requires no dedicated hardware, lowering the user threshold, and achieves cross-platform compatibility based on the web technology stack. Through lightweight front-end algorithms and rendering optimization, it ensures smooth operation even on low-configuration devices, significantly improving the real-time performance, immersion, and realism of the interaction, providing efficient, convenient, and expressive technical support for application scenarios such as virtual try-on and online social networking. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram illustrating an application scenario of the web-based front-end interaction method provided in the embodiments of this application; Figure 2 This is a flowchart of an interaction method based on a web front-end according to an embodiment of this application; Figure 3 This is a structural block diagram of a web-based interactive system according to one embodiment of the present application; Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0022] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0023] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0025] In this document, it should be understood that the terms used may be technical means used to implement part of this invention or other summary technical terms. For example, the terms may include: WebGL (Web Graphics Library) refers to a JavaScript-based 3D graphics standard used to implement hardware-accelerated 3D rendering in browsers.

[0026] API (Application Programming Interface): refers to the interaction interface between software components, such as the interface used to call the camera and process visual data in this article.

[0027] Virtual Character: A digital avatar built based on a 3D model, which can simulate the posture and expression of a real user through motion mapping.

[0028] Motion capture refers to the process of extracting user's limb, facial, or hand movement data using computer vision technology. In this article, it specifically refers to non-invasive motion capture based on monocular or binocular cameras.

[0029] Attitude estimation refers to the technique of using algorithms to predict the position and angle of key human body nodes (such as joints) from images and generate skeletal structure data.

[0030] Virtual Goods: refers to digital 3D items (such as backpacks, watches, hats, etc.) used to interact with virtual avatars, and have collision detection and attachment logic with virtual avatars.

[0031] Mapping Algorithm: A method for calculating parameters that convert real user motion data into skeletal animation parameters for a virtual avatar, ensuring that the virtual avatar's movements are consistent with the user's movements.

[0032] Gltf (GL Transmission Format): refers to an efficient 3D model format suitable for model loading and rendering in a web environment.

[0033] This application provides a web-based interactive method applicable to scenarios such as virtual try-on, online social networking, and digital twin displays. This method aims to achieve motion capture and real-time virtual interaction with relatively low-threshold hardware requirements.

[0034] Figure 1 This is a schematic diagram illustrating an application scenario of the web-based front-end interaction method provided in the embodiments of this application, such as... Figure 1 As shown, the solution in this application can be applied to an interactive scenario entirely based on a web browser environment. In this application environment, the user uses a terminal device with a built-in camera, accesses a service page through a browser, and grants camera permissions. The terminal device captures a video stream containing the user in real time and processes the video stream data in the browser frontend.

[0035] The terminal device generates a basic virtual avatar based on the user's image using a Gaussian virtual human algorithm. Simultaneously, a computer vision model deployed at the front end captures the user's limb, facial, and hand movements from the video stream in real time, extracting 3D keypoint coordinates. This coordinate data is then processed through smoothing filtering and inverse kinematics (IK) algorithms to convert it into skeletal animation parameters for the virtual avatar, enabling it to perform natural, synchronized movements with the user in real time. When the user interacts with virtual objects (such as backpacks or hats), the relative position and posture between the virtual object and the virtual avatar are dynamically calculated and adjusted based on the user's specific actions (such as turning or raising a hand to grab). Finally, the driven virtual avatar and the dynamically interacting virtual objects are synthesized in real time, and the final interactive scene is visualized on the user's terminal screen. The entire process has extremely low latency, ensuring a smooth and immersive experience.

[0036] exist Figure 1 In the application scenarios shown, the terminal device can be any device with a regular camera and browser, such as a personal computer, smartphone, tablet, or VR all-in-one device, without the need to install additional applications or equip it with dedicated sensors.

[0037] Figure 2 This is a flowchart illustrating a web front-end-based interactive method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: S201 uses the browser to access the terminal camera, acquires the user's video stream, and processes the video stream in the browser front-end to extract key points of the human body.

[0038] In this embodiment, this step aims to achieve non-intrusive capture of user motion data by using a standard web technology and the user terminal's built-in ordinary camera as the sole input device.

[0039] Specifically, firstly, under an HTTPS page that conforms to security protocols, the browser's application programming interface (API), such as the get User Media API, is used to call the front or rear camera of the terminal device (such as a personal computer, smartphone, or tablet) to obtain the user's real-time video stream.

[0040] Optionally, to balance clarity and processing efficiency, the video stream resolution can be set to 1280x720 and the frame rate to 30fps.

[0041] Furthermore, after acquiring the video stream, in order to achieve cross-platform operation and avoid delays caused by complex backend data transmission and processing, this embodiment deploys the visual recognition and key point extraction model in the browser front-end environment for execution.

[0042] Specifically, consecutive image frames from the video stream are concurrently forwarded to multiple visual detection models deployed in the browser environment, including pose detection models (such as Media Pipe Pose), hand keypoint detection models (such as MediaPipe Hands), and face keypoint detection models (called via Face-API).

[0043] Additionally, it should be noted that in this embodiment, to ensure the smoothness of the UI and avoid visual computation blocking the main thread, the model's computational tasks are executed in a Web Worker thread in the browser's background. Each model processes image frames concurrently, extracting pose keypoints (e.g., 33 limb key nodes), hand keypoints (e.g., 21 hand joint nodes), and facial keypoints (e.g., 68 facial feature points) from the images, collectively forming a complete set of human keypoint data. The extracted keypoint coordinates are initially in an image-based coordinate system, which needs to be further transformed to a world coordinate system centered on the user's hip, providing a unified three-dimensional spatial reference for subsequent motion mapping.

[0044] Step S201 uses a browser front-end visual model to capture key points of the whole body, hands, and face in real time using a regular camera, and leverages Web Worker to ensure smooth processing. This achieves low-hardware-cost, cross-platform, and refined motion data acquisition, laying the foundation for subsequent natural interaction.

[0045] S202 uses a layered motion mapping engine to map human body key points to the skeletal animation parameters of the virtual avatar.

[0046] This step aims to convert the potentially noisy discrete keypoint data extracted from the front end into smooth, natural virtual avatar skeletal animation parameters that conform to human kinematics. In this embodiment, a layered motion mapping engine is used to implement this conversion process.

[0047] Specifically, at the data processing level, in order to solve the problem of key point data jitter caused by changes in light, occlusion or rapid movement during camera capture, a Kalman filter algorithm is used to smooth the original human key point data.

[0048] The following iterative process of prediction and update is used to reduce data noise and control the overall action delay within 100ms: Predicted value = Actual value of the previous frame + speed × time difference Actual value = Predicted value + gain × (current detection value - predicted value); In the middle layer of the algorithm, the smoothed human keypoint data is converted into the rotation angle of the corresponding skeletal nodes of the virtual image through the inverse kinematics (IK) algorithm, thus obtaining the skeletal animation parameters.

[0049] For example, by taking the three-dimensional keypoint positions of the user's shoulder, elbow, and wrist as input, the IK algorithm can calculate the angles that the shoulder, elbow, and wrist joints in the virtual avatar's arm need to rotate, so that the end of the virtual avatar's hand can accurately reach the spatial position corresponding to the user's hand, ensuring the naturalness and physical realism of the virtual avatar's movements.

[0050] At the logical level, a state machine manages the macroscopic action states of the virtual avatar, with multiple preset action states, such as "still state," "turning state," and "grabbing state." This state machine outputs state switching instructions based on skeletal animation parameters and preset judgment thresholds.

[0051] For example, the system continuously monitors the user's spinal rotation angle calculated from key points on the human body. When this rotation angle exceeds a preset threshold (e.g., 30°), the state machine outputs a command to switch from a "stationary state" to a "turning state." Simultaneously, it calculates the 3D spatial distance between the user's hand key points and virtual objects in the scene in real time. When this distance is less than a preset distance threshold (e.g., 30 pixels), it triggers and outputs a command to switch to a "grabbing state." To handle complex actions, each state switching command can be configured with priority information. When multiple state switching trigger conditions are met simultaneously, only the command with the highest priority is output based on the priority information, avoiding action conflicts.

[0052] Step S202 transforms the original key points into smooth, natural, and logically clear virtual image animation parameters through filtering, inverse dynamics, and state machine layered processing. These parameters are used for the interaction of virtual objects in subsequent interactions, achieving low-latency and high-fidelity motion mapping and significantly improving the realism of the virtual image.

[0053] S203, calculate the relative pose of the virtual item and the virtual image based on the preset attributes and skeletal animation parameters of the virtual item.

[0054] To achieve natural and scalable interaction between virtual items and virtual avatars, this embodiment designs an adaptive virtual item interaction framework. First, by parsing the JSON configuration file corresponding to each virtual item, its preset attributes are obtained. These attributes define key information about the item, such as item type (e.g., wearable item), attachment point (e.g., back, shoulder, wrist), and physical characteristics (e.g., collision radius).

[0055] Specifically, during pose calculation, the real-time rotation angle of a specific bone (such as the spinal bone) of the virtual character is obtained based on the skeletal animation parameters output by S102. Based on this rotation angle and the attachment point information defined in the item configuration file, the relative following pose of the virtual item with respect to the pose key points can be calculated.

[0056] For example, when the attachment point of a "shoulder bag" is defined as "left shoulder", even if the user's virtual avatar turns around, the shoulder bag model will adjust synchronously according to the pose change of the virtual avatar's shoulder bones to maintain the visual effect of it always hanging on the shoulder.

[0057] Furthermore, to achieve dynamic interaction, the technical solution provided in this embodiment continuously detects whether the user's hand keypoints overlap with the collision radius of the virtual object. When overlap is detected, such as when the user reaches out to touch a backpack, the attachment relationship of the virtual object is dynamically switched to the user's hand keypoints. Subsequently, based on the skeletal animation parameters of the hand and the preset attributes of the object, the relative following pose of the virtual object relative to the hand keypoints is recalculated to achieve virtual grasping and movement effects.

[0058] Step S203 simplifies the adaptation process for new items through standardized JSON configuration and a dynamic attachment algorithm, solving the problems of developing separate logic for different items and high adaptation costs in existing solutions. Simultaneously, by dynamically calculating the item's following pose and switching attachment relationships based on collision detection, it enables natural interaction between virtual items and virtual avatars based on the user's real actions, supporting complex logic such as turning and following, and dynamic grabbing, greatly enhancing the immersiveness, realism, and scalability of the interaction.

[0059] S204, in a browser environment, renders and displays interactive screens of virtual items and virtual avatars based on their relative poses.

[0060] The purpose of this step is to efficiently present the dynamic relationship between the aforementioned calculated virtual images and virtual items on the user's browser interface, ensuring the real-time performance and smoothness of the interactive screen. In this embodiment, optionally, the rendering system is built based on the Three.js and WebGL technology stack.

[0061] It should also be noted that several optimization measures were taken to improve performance and loading speed. First, the 3D models were lightweighted, using the Gltf format to store the models of virtual avatars and items, and the Draco compression algorithm was used to compress the geometric data of the models at a high rate, reducing the model file size by more than 50%. Second, to accelerate secondary loading and support offline access, the 3D models and texture resources of virtual avatars and items were cached using Service Worker technology on the browser side.

[0062] During the rendering loop, the rendering module synthesizes the final interactive screen in real time based on the latest relative poses of the virtual image and virtual objects calculated by S103, as well as the 3D models and texture resources loaded from Service Worker or the network, and ensures that the rendering frame rate is not less than 30fps, providing users with smooth, low-latency visual feedback.

[0063] Step S204 utilizes front-end optimization technologies such as model compression, hierarchical rendering, and resource caching to achieve high-performance real-time 3D rendering in a standard browser environment, ensuring a smooth and high-quality virtual interactive visual experience even on low-configuration devices.

[0064] Furthermore, to further enrich the details and realism of the interaction, the method in this embodiment also includes: pre-defining one or more variable presentation states of the virtual item in the JSON configuration file corresponding to the virtual item, based on its physical characteristics, and the state switching trigger conditions corresponding to each state. During operation, it is determined in real time whether the interactive actions generated by the user's key points satisfy the aforementioned preset state trigger conditions. If it is determined that they are satisfied, the rendering attributes of the virtual item's 3D model will be updated accordingly based on the trigger conditions, and / or specific animation effects bound to it will be triggered.

[0065] For example, for a virtual backpack item, an "open" state can be defined in its JSON file, triggered by "the collision radius of the user's other hand's key point overlapping with the backpack's buckle model." When this "open" action is detected in real time, the backpack model's preset "open" animation is triggered, and the rendering properties of its internal materials are updated from invisible to visible, thus simulating a realistic backpack opening effect.

[0066] Furthermore, to achieve more complex inter-item interaction (i.e., one item affects another item), the method in this embodiment also includes configuring linkage triggering conditions for the first virtual item (such as "key") and the second virtual item (such as "treasure chest") in a JSON configuration file.

[0067] For example, after a user picks up a "key" in a "grab" state, the key's collider, driven by the user's hand's key points, is moved near the lock cylinder collider of the "treasure chest." When the system detects in real time that the two meet preset linkage trigger conditions (e.g., colliders overlap and the key model's rotation angle meets a specific threshold), a related animation effect will be triggered. First virtual item (key): Play the "rotate insertion" animation and update its attachment point so that it "attaches" to the lock cylinder; Second virtual item (treasure chest): Its lock cylinder model's rendering attribute is updated to "invisible", and at the same time, its "lid opens" animation is triggered, revealing the internal model.

[0068] This configuration enables realistic interaction between virtual items, greatly enriching the depth and fun of the interaction.

[0069] Through steps S101 to S103 above, this solution solves the problems of high hardware dependence, poor cross-platform compatibility, simple interaction links, and severe latency and distortion in existing technologies. It eliminates the need for dedicated hardware, lowering the barrier to entry for users, and achieves cross-platform compatibility based on the Web technology stack. Through lightweight front-end algorithms and rendering optimization, it ensures smooth operation even on low-configuration devices, significantly improving the real-time performance, immersion, and realism of the interaction, providing efficient, convenient, and expressive technical support for application scenarios such as virtual try-on and online social networking.

[0070] On the other hand, embodiments of this application also provide an interactive system based on a web front-end, which is used to implement the above-described method. Figure 3 This is a structural block diagram of a web-based interactive system according to one embodiment of the present application, such as... Figure 3 As shown, the system includes: a data acquisition module 30, an action processing module 31, and a rendering module 32.

[0071] The acquisition module 30 is used to access the terminal camera through the browser, acquire the user's video stream, and process the video stream in the browser front end to extract human body key points.

[0072] In this embodiment, this step aims to achieve non-intrusive capture of user motion data by using a standard web technology and the user terminal's built-in ordinary camera as the sole input device.

[0073] Specifically, firstly, under an HTTPS page that conforms to security protocols, the browser's application programming interface (API), such as the get User Media API, is used to call the front or rear camera of the terminal device (such as a personal computer, smartphone, or tablet) to obtain the user's real-time video stream. Optionally, to balance clarity and processing efficiency, the video stream resolution is set to 1280x720 and the frame rate is set to 30fps.

[0074] Furthermore, after acquiring the video stream, in order to achieve cross-platform operation and avoid delays caused by complex backend data transmission and processing, this embodiment deploys the visual recognition and key point extraction model in the browser front-end environment for execution.

[0075] The acquisition module 30 uses a browser front-end visual model to capture key points of the whole body, hands, and face in real time using a regular camera, and leverages Web Worker to ensure smooth processing. This achieves low-hardware-cost, cross-platform, and refined motion data acquisition, providing a foundation for subsequent natural interaction.

[0076] The motion processing module 31 is used to map human key points to the skeletal animation parameters of the virtual image through a layered motion mapping engine; and to calculate the relative pose of the virtual object and the virtual image based on the preset attributes and skeletal animation parameters of the virtual object.

[0077] This step aims to convert the potentially noisy discrete keypoint data extracted from the front end into smooth, natural virtual avatar skeletal animation parameters that conform to human kinematics. In this embodiment, a layered motion mapping engine is used to implement this conversion process.

[0078] Specifically, at the data processing level, in order to solve the problem of key point data jitter caused by changes in light, occlusion or rapid movement during camera capture, a Kalman filter algorithm is used to smooth the original human key point data.

[0079] In the middle layer of the algorithm, the smoothed human keypoint data is converted into the rotation angle of the corresponding skeletal nodes of the virtual image through the inverse kinematics (IK) algorithm, thus obtaining the skeletal animation parameters.

[0080] At the logical level, a state machine manages the macroscopic action states of the virtual avatar, with multiple preset action states, such as "still state," "turning state," and "grabbing state." This state machine outputs state switching instructions based on skeletal animation parameters and preset judgment thresholds.

[0081] Furthermore, to achieve natural and scalable interaction between virtual items and virtual avatars, this embodiment designs an adaptive virtual item interaction framework. First, by parsing the JSON configuration file corresponding to each virtual item, its preset attributes are obtained. These attributes define key information about the item, such as item type (e.g., wearable item), attachment point (e.g., back, shoulder, wrist), and physical characteristics (e.g., collision radius).

[0082] During pose calculation, skeletal animation parameters are used to obtain the real-time rotation angles of specific bones (such as the spine) of the virtual character. Based on these rotation angles and the attachment point information defined in the item configuration file, the relative following pose of the virtual item with respect to the pose keypoints can be calculated.

[0083] The motion processing module 31 transforms the original key points into smooth, natural, and logically clear virtual image animation parameters through filtering, inverse dynamics, and state machine layered processing, which are used for the interaction of virtual objects in subsequent interactions. Furthermore, through standardized JSON configuration and dynamic attachment algorithm, the adaptation process of new items is simplified. By dynamically calculating the item's following pose and switching the attachment relationship based on collision detection, natural interaction between virtual items and virtual images based on the user's real actions is realized, improving the immersion, realism, and scalability of the interaction.

[0084] The rendering module 32 is used to render and display interactive screens of virtual items and virtual images in a browser environment based on their relative poses.

[0085] The purpose of the rendering module 32 is to efficiently present the dynamic relationship between the aforementioned calculated virtual images and virtual items on the user's browser interface, ensuring the real-time performance and smoothness of the interactive screen. In this embodiment, the rendering system is built on the Three.js and WebGL technology stack. By comprehensively utilizing front-end optimization technologies such as model compression, hierarchical rendering, and resource caching, high-performance real-time 3D rendering is achieved in a standard browser environment, ensuring a smooth and high-quality virtual interactive visual experience even on low-configuration devices.

[0086] This system addresses the problems of high hardware dependence, poor cross-platform compatibility, simple interaction links, and severe latency and distortion inherent in existing technologies. It eliminates the need for dedicated hardware, lowering the barrier to entry, and achieves cross-platform compatibility based on the Web technology stack. Through lightweight front-end algorithms and rendering optimizations, it ensures smooth operation even on low-configuration devices, significantly improving the real-time performance, immersion, and realism of the interaction. This provides efficient, convenient, and expressive technical support for applications such as virtual try-on and online social networking.

[0087] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operating system to run, the computer programs are executed by the processor to implement a web-based front-end interaction method, and the database stores data.

[0088] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to computer program instructions. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A Web front-end based interaction method, characterized in that, The interaction method comprises: Calling a terminal camera through a browser, acquiring a video stream of a user, and processing the video stream at a browser front end to extract human body key points; Mapping the human body key points to skeletal animation parameters of a virtual image through a layered action mapping engine; According to preset attributes of a virtual object and the skeletal animation parameters, calculating a relative pose of the virtual object and the virtual image; In a browser environment, rendering and displaying an interaction picture of the virtual object and the virtual image according to the relative pose.

2. The interaction method of claim 1, wherein, Mapping the human body key points to skeletal animation parameters of a virtual image through a layered action mapping engine comprises: At a data processing bottom layer, using a Kalman filtering algorithm to perform smoothing processing on the human body key point data; At an algorithm middle layer, converting the human body key point data after the smoothing processing into rotation angles of corresponding skeletal nodes of the virtual image through a reverse dynamics algorithm, to obtain the skeletal animation parameters, wherein: At a logic upper layer, presetting a plurality of action states through a state machine, and outputting state switching instructions according to the skeletal animation parameters and a preset determination threshold.

3. The interaction method of claim 2, wherein, The action states comprise a static state, a turning state and a grabbing state, and outputting the state switching instructions according to the skeletal animation parameters and the preset determination threshold comprises: According to the human body key points, acquiring a spinal skeletal rotation angle of the user, and when the rotation angle is greater than a preset angle threshold, outputting a state switching instruction switching from the static state to the turning state; According to the human body key points, calculating a real-time spatial distance between a hand of the user and the virtual object, and when the spatial distance is less than a preset distance threshold, outputting a state switching instruction switching from a current state to the grabbing state; Wherein, each state switching instruction is configured with priority information, and when a plurality of state switching trigger conditions are simultaneously met, the state switching instructions are output according to the priority information.

4. The interaction method of claim 1, wherein, Processing the video stream at the browser front end to extract human body key points comprises: Under an HTTPS page, acquiring the video stream through a browser application programming interface; Forwarding image frames in the video stream to a posture detection model, a hand key point detection model and a face key point detection model deployed in a browser environment; Concurrently processing the image frames through the posture detection model, the hand key point detection model and the face key point detection model executed in a Web Worker thread in the background of the browser, to respectively extract posture key points, hand key points and face key points in the image frames, to obtain the human body key points.

5. The interaction method of claim 4, wherein, Calculating the relative pose of the virtual object and the virtual image comprises: Acquiring preset attributes of the virtual object by analyzing a JSON configuration file corresponding to the virtual object, wherein the preset attributes comprise an object type, an attachment point and physical properties; According to the skeletal animation parameters, acquiring a rotation angle of a spinal skeleton of the virtual image, and based on the rotation angle and the preset attributes, calculating a relative following pose of the virtual object relative to the posture key points; detect whether the hand key point of the user and the collision radius of the virtual object overlap, if so, switch the attachment relationship of the virtual object to the hand key point, and calculate the relative following pose of the virtual object relative to the hand key point based on the skeleton animation parameters and the preset attributes.

6. The interaction method of claim 5, wherein, The method further comprises: In the JSON configuration file corresponding to the virtual object, define the variable presentation state of the virtual object and the corresponding trigger condition according to the physical properties; Determine whether the interactive action generated by the human body key point meets the trigger condition; If so, update the rendering attributes of the three-dimensional model of the virtual object according to the trigger condition, and / or trigger the animation effect corresponding to the three-dimensional model.

7. The interaction method of claim 6, wherein, The method further comprises: In the JSON configuration file, configure the linkage trigger condition of the first virtual object and the second virtual object; Detect whether the collision body of the first virtual object and the second virtual object meets the linkage trigger condition; If so, update the rendering attributes of the three-dimensional model of the first virtual object and the second virtual object, and / or trigger the associated animation effect corresponding thereto.

8. The interaction method of claim 1, wherein, The method further comprises: Using the Draco compression algorithm, perform geometric data compression on the three-dimensional models of the virtual avatar and the virtual object, wherein the three-dimensional models of the virtual avatar and the virtual object are stored in Gltf format; Cache the three-dimensional models of the virtual avatar and the virtual object and the texture resources through the Service Worker; In a browser environment, according to the relative pose of the virtual object and the virtual avatar, and the 3D models and texture resources of the virtual avatar and the virtual object loaded offline from the Service Worker, render and display the interactive screen of the virtual object and the virtual avatar.

9. A web front-end based interactive system, characterized in that, The interaction system comprises a collection module, an action processing module and a rendering module, wherein: The collection module is configured to call the terminal camera through the browser to obtain the video stream of the user, and process the video stream in the browser front end to extract the human body key point; The action processing module is configured to map the human body key point to the skeleton animation parameters of the virtual avatar through a hierarchical action mapping engine; And according to the preset attributes of the virtual object and the skeleton animation parameters, calculate the relative pose of the virtual object and the virtual avatar; The rendering module is configured to render and display the interactive screen of the virtual object and the virtual avatar according to the relative pose in a browser environment.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 8. The processor executes the computer program to implement the method of any one of claims 1 to 8.

Images