Method for controlling a virtual character in a game and device for controlling a virtual character in a game

Abstract

The application discloses a virtual character control method and device in a game, and the method comprises the following steps: obtaining somatosensory input data of a target object, determining first skeleton posture data of the target object according to the somatosensory input data, mapping the first skeleton posture data to a virtual character skeleton of a target virtual character in a target game to obtain second skeleton posture data, identifying a virtual action to be executed by the target virtual character based on the second skeleton posture data to obtain a plurality of candidate virtual actions, determining the target virtual action based on the plurality of candidate virtual actions, action control delay information and virtual character motion constraints corresponding to the target game, and controlling the target virtual character to execute the target virtual action. In this way, the application enables a player to complete real-time control of a virtual character in a game through body movements, and the interactive immersion is stronger. Moreover, the structural difference between different skeleton systems can be reduced, and joint distortion and abnormal dressing can be avoided.

Description

Technical Field

[0001] This application relates to the field of game technology, specifically to a method and device for controlling virtual characters in a game. Background Technology

[0002] In traditional motion-sensing game character control methods, the system collects the player's motion input data and converts it into corresponding movements for the virtual character, allowing the player to control the virtual character through body postures. However, if the converted virtual character movements do not meet the player's expectations, the virtual character is prone to abnormalities such as posture deviation, misjudgment of movements, or violation of game movement rules because the player's skeleton and the game character's skeleton are two different systems, and the movement judgment lacks candidate filtering and constraint verification mechanisms. Players need to repeatedly adjust their limbs to obtain ideal feedback, resulting in inconvenient control, low accuracy, and a poor interactive experience. Summary of the Invention

[0003] This application provides a method and device for controlling virtual characters in a game. By mapping the skeleton posture corresponding to the motion input data to the skeleton of the virtual character, and identifying multiple candidate virtual actions based on the mapping results, the target virtual action is determined by combining motion control delay information and virtual character motion constraints for control. This enables the virtual character to accurately respond to the player's motion operation and conform to the game's motion rules, thereby improving the accuracy and responsiveness of motion control.

[0004] On one hand, embodiments of this application provide a method for controlling a virtual character in a game, the method comprising: Acquire the motion input data of the target object; Based on the somatosensory input data, the first skeleton pose data of the target object is determined; The first skeleton pose data of the target object is mapped to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton pose data of the target virtual character. Based on the second skeleton pose data, the virtual actions to be performed by the target virtual character are identified, and multiple candidate virtual actions are obtained; Based on the multiple candidate virtual actions, action control delay information, and the motion constraints of the virtual character corresponding to the target game, a target virtual action is determined, wherein the action control delay information is used to indicate the delay between the target object performing the action and the target virtual character responding to the target object performing the action; Control the target virtual character to perform the target virtual action.

[0005] On the other hand, embodiments of this application provide a virtual character control device in a game, the device comprising: The acquisition module is used to acquire the motion input data of the target object; An object pose determination module is used to determine the first skeleton pose data of the target object based on the somatosensory input data. The character pose determination module is used to map the first skeleton pose data of the target object to the virtual character skeleton of the target virtual character in the target game, so as to obtain the second skeleton pose data of the target virtual character. The first action recognition module is used to identify the virtual action to be performed by the target virtual character based on the second skeleton posture data, and obtain multiple candidate virtual actions; The second action recognition module is used to determine the target virtual action based on the multiple candidate virtual actions, action control delay information and the motion constraints of the virtual character corresponding to the target game, wherein the action control delay information is used to indicate the delay between the target object performing the action and the target virtual character responding to the target object performing the action; The control module is used to control the target virtual character to perform the target virtual action.

[0006] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the virtual character control method in a game as described in any of the above embodiments.

[0007] On the other hand, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the virtual character control method in the game as described in any of the above embodiments by calling the computer program stored in the memory.

[0008] On the other hand, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the virtual character control method in a game as described in any of the above embodiments.

[0009] The virtual character control method, virtual character control device, computer-readable storage medium, electronic device, and computer program product provided in this application can acquire the motion input data of a target object, determine the first skeleton posture data of the target object based on the motion input data, map the first skeleton posture data of the target object to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton posture data of the target virtual character, identify the virtual action to be performed by the target virtual character based on the second skeleton posture data to obtain multiple candidate virtual actions, determine the target virtual action based on the multiple candidate virtual actions, action control delay information, and the motion constraints of the virtual character corresponding to the target game, and control the target virtual character to perform the target virtual action. This allows players and other target objects to complete real-time control of the game virtual character solely through body movements, resulting in a stronger sense of immersion compared to traditional gamepad button control. Furthermore, by adapting and converting the real human posture into the virtual character posture through skeleton mapping, the structural differences between different skeleton systems can be reduced, avoiding joint distortion and clipping anomalies. Meanwhile, by using multi-candidate action recognition and multi-constraint joint decision-making to determine the target action, the delay between the player's action and the virtual character's response, as well as the virtual character's motion constraints, can all participate in the target action decision-making, reducing the probability of misjudgment and avoiding conflicts between independent optimization of each module to a certain extent. Thus, while ensuring the responsiveness of the motion control, it can also ensure the robustness and compliance of the action output. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is one of the flowcharts illustrating a virtual character control method in a game provided in an embodiment of this application.

[0012] Figure 2 This is a second flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0013] Figure 3 This is the third flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0014] Figure 4 The fourth flowchart illustrates the virtual character control method in a game provided in this application embodiment.

[0015] Figure 5The fifth flowchart illustrates the virtual character control method in a game provided in this application embodiment.

[0016] Figure 6 This is the sixth flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0017] Figure 7 This is one of the application scenarios of the virtual character control method in a game provided in the embodiments of this application.

[0018] Figure 8 This is the seventh flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0019] Figure 9 This is the eighth flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0020] Figure 10 This is the ninth flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0021] Figure 11 This is a second schematic diagram illustrating an application scenario of the virtual character control method in a game provided in this application embodiment.

[0022] Figure 12 This is the tenth flowchart illustrating the virtual character control method in a game provided in this application embodiment.

[0023] Figure 13 A schematic diagram of the structure of a virtual character control device in a game provided in an embodiment of this application.

[0024] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This application provides a method for controlling virtual characters in a game, a device for controlling virtual characters in a game, a computer-readable storage medium, an electronic device, and a computer program product. Specifically, the method for controlling virtual characters in a game according to this application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart TV, wearable smart device, smart vehicle terminal, etc. The terminal can also include a client, which can be a browser client, instant messaging client, or mini-program, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0027] It should be noted that, in this embodiment, the execution entity of the virtual character control method in the game can be a terminal device or a server. The terminal device can be a local terminal device or a client device in a cloud platform. This embodiment does not limit the type of execution entity.

[0028] For example, when the virtual character control method in the game runs on a terminal device, the terminal device may include a display screen and a processor. The display screen is used to present a graphical user interface (GUI) and receive instructions generated by the user interacting with the GUI. The processor is used to store applications, generate the GUI, respond to instructions, and control the display of the GUI on the display screen. When the user operates the GUI through the display screen, the GUI can control the local content of the terminal device in response to the received operation instructions. The terminal device can provide the GUI to the user in various ways, such as rendering it on the terminal device's display screen or presenting the GUI through holographic projection.

[0029] For example, when the virtual character control method in the game runs on a server, this method can be implemented and executed based on a cloud system. The cloud system includes servers and client devices. The application's runtime and the graphical user interface (GUI) presentation are separate. The storage and execution of the virtual character control method in the game are completed on the server. The GUI presentation, however, is completed on the client. The client is mainly used for receiving and sending data and for presenting the GUI. For example, the client can be a display device with data transmission capabilities located close to the user, such as a mobile terminal, television, computer, PDA, personal digital assistant, or head-mounted display device. However, the terminal device for data processing is the server in the cloud. During this process, the user operates the client to send instructions to the server. The server executes the instructions, encodes and compresses the GUI data, returns it to the client via the network, and finally, the client decodes and outputs the GUI.

[0030] It should be noted that, in this embodiment, the executing entity of the virtual character control method in the game can be a terminal device or a server. The terminal device can be a local terminal device or a client device in the aforementioned cloud system. This embodiment does not limit the type of executing entity.

[0031] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0032] The fundamental requirement of motion-sensing games is to achieve real-time synchronization between player movements and game characters; that is, after a player makes a physical movement, the game character can immediately and accurately reproduce the corresponding movement. Transforming this fundamental requirement into a stable online product involves the collaborative efforts of multiple devices, systems, and stages, with engineering complexity far exceeding the scope of optimizing a single algorithm. The complete chain of motion-sensing interaction encompasses multiple stages, including data acquisition, data cleaning, posture inference, skeleton redirection, rule validation, network synchronization, animation blending, and rendering submission. Delays or deviations in any stage will directly impact the final interactive experience.

[0033] At the input data level, motion-sensing projects typically require simultaneous access to multiple heterogeneous data sources, including skeletal keypoint data captured by cameras, posture data output by inertial measurement units (IMUs), controller button inputs, and data from mobile auxiliary sensors. These different data sources vary in sampling frequency, timestamp accuracy, and data loss patterns. Camera data is susceptible to lighting conditions and limb occlusion, IMUs suffer from cumulative drift, and controller inputs only provide discrete button status information. The challenge in development is not the accuracy of individual data streams, but rather the synchronization and alignment of multiple data streams over time. If the timelines are not unified, it will cause jumps or abrupt changes in character movements, resulting in players perceiving lag in response or deviations from expected actions.

[0034] In the skeletal mapping stage, the generic skeleton output from human pose estimation and the bound skeleton of the game character belong to two different systems. The former is based on the definition of generic human keypoints, while the latter is deeply bound to art assets, equipment attachment points, and skill effect anchor points. The generic skeleton and the bound skeleton differ in bone segment proportions, joint degrees of freedom, and naming hierarchy. While the mapping effect is generally acceptable in static poses, it easily leads to problems such as elbow backward bending, model penetration, foot slippage, and misalignment of weapons and hands during large-scale dynamic movements. In multiplayer battles, these deviations further affect the accuracy of hit detection, thereby undermining competitive fairness.

[0035] In terms of gameplay design, player movements are continuous analog signals, while game skill systems typically operate based on discrete state events. Taking a striking action as an example, a complete action includes three consecutive phases: preparation, power generation, and recovery. If the system judges the action solely based on instantaneous angle thresholds, it may easily misinterpret the preparation phase as a light strike or the recovery phase as the start of the next action. Such recognition biases directly impact the player's experience in competitive scenarios. Especially during matches, skill trigger failures significantly reduce player satisfaction, thereby affecting user retention and monetization conversion.

[0036] Performance and network factors constitute another layer of constraints. Specifically, the individual latency of each link in the entire motion-sensing interaction chain may only be a few tens of milliseconds, but the cumulative latency often approaches or exceeds the human-perceptible latency threshold. The online operating environment has many uncontrollable factors, including home network jitter, performance degradation of mobile devices due to overheating, and background processes consuming computing resources. The same algorithm performs significantly differently under different hardware configurations and network environments. If the development team only focuses on the recognition accuracy in offline environments and ignores the long-tail performance of end-to-end latency and jitter distribution, the product will struggle to adapt to the complexity of real-world user environments after launch.

[0037] In terms of engineering collaboration, motion-sensing projects involve the collaboration of multiple teams, including algorithm developers, animation engineers, technical artists, client developers, server developers, combat planners, quality testers, and operations teams. Without unified measurement standards and coordination mechanisms, localized optimizations can easily lead to a decline in overall performance: increasing recognition sensitivity by the algorithm team increases the compensation pressure on the client; expanding the filtering window to improve smoothness delays combat judgment timing; and tightening trigger thresholds to ensure fairness reduces the freedom of motion-sensing controls. Independent optimization by each team may result in no substantial improvement to the overall experience.

[0038] During the product operation phase, with each version iteration, new maps, characters, skins, weapons, and action packs are continuously added to the game, gradually reducing the compatibility of existing system parameters. Trigger thresholds that are stable in the testing environment may shift after the action resource library is updated. If the motion sensing system lacks online data monitoring, canary release, and rapid rollback mechanisms, technical debt will continue to accumulate during version iterations, ultimately leading to issues where the demo version performs well but the online version lacks stability.

[0039] The implementation solutions in related technologies can be divided into four categories, and most projects adopt a combination of multiple solutions. These solutions can all reach a usable state in the prototype verification stage, but their limitations will gradually become apparent in long-term stable operation scenarios.

[0040] The first type is based on rule-based thresholds and fixed mapping. This approach calculates features such as the angle or velocity of key skeletal points. When these values ​​exceed a preset threshold, the corresponding action is identified, and a preset redirection matrix drives the character's movement. The advantages of rule-based thresholds and fixed mapping are short development cycles, intuitive debugging, and low computational resource consumption, making it suitable for rapid verification in the early stages of a project. However, it has limitations in adapting to individual player differences; variations in height, arm span, range of motion, and movement habits can all affect the accuracy of threshold determination. A lenient threshold setting can lead to false triggers, while a strict one can result in missed detections. Fixed mapping performs reasonably well in static poses, but it is prone to joint distortion and foot drift when dealing with high-speed movements. In later stages of a project, significant manual correction logic is often required for specific characters or actions, leading to continuously increasing maintenance costs.

[0041] The second type is a solution based on offline motion capture data and the engine's state machine. This approach standardizes motion resources during the production phase, and then primarily performs motion segment blending and state transitions after deployment. This model works well in traditional non-motion-sensing games because controller input is inherently discrete signals. However, when applied to real-time motion-sensing input, a granularity mismatch exists between the player's continuously changing motion flow and the discretely triggered events of the state machine. To reduce the probability of misjudgments, teams typically add transition states and guard frames, at the cost of increased motion response latency. Players will perceive smooth animation effects but less responsive controls.

[0042] The third category is end-to-end prediction and direct action output solutions. These solutions attempt to bypass manual rule design and directly predict character pose sequences from the input sequence, offering a high degree of automation and good performance on academic metrics. However, they suffer from several problems in practical game applications: the prediction results may not conform to combat rules, such as incorrect attack order; they may not meet physical collision and foot constraints, such as feet penetrating the ground; and they lack temporal stability under network jitter, resulting in frame skipping. To address these issues, projects typically require adding rule correction and smoothing modules in subsequent stages, leading to a layered system structure, a rapidly expanding parameter space, and decreased maintainability.

[0043] The fourth type is a network synchronization scheme that combines client-side prediction with server-side correction. This scheme is the standard synchronization architecture for multiplayer games, aiming to reduce the perceived latency of actions by players. The problem is that if the front-end action semantic recognition itself is unstable, server-side correction operations will be triggered frequently. Players do not see smooth state corrections, but rather intermittent jumps in character actions. To mitigate the degree of these jumps, the client expands the interpolation window, but this weakens the real-time performance of the actions, ultimately leading to a cycle where accuracy and latency are difficult to balance.

[0044] From an engineering architecture perspective, the four solutions mentioned above share three common structural flaws. First, they lack a unified time base. Most solutions assume that data sources are naturally synchronized, but in real-world online environments, time drift and data out of order are common occurrences, not anomalies. The lack of a unified time base means that all subsequent action understanding stages are built on an unstable foundation. Second, they lack a unified semantic intermediate layer. There is no stable conversion layer between human action semantics and game skill semantics. The system can only establish a direct mapping between raw posture data and skill identifiers, leading to poor interpretability, difficulty in parameter adjustment, and high costs for cross-version migration. Third, they lack unified optimization goals. Visual naturalness, rule validity, network consistency, and device fairness are usually optimized separately, leading to conflicts between local optima. This often manifests as a minor version fixing one type of problem but exacerbating another, forcing the team to repeatedly weigh multiple metrics.

[0045] In summary, the various solutions proposed in the relevant technologies are not entirely unusable, but they are difficult to simultaneously meet the comprehensive requirements of high real-time performance, high stability, high fairness, and operability, and to maintain stable operation in long-term iterations.

[0046] For the above issues, please refer to Figure 1 , Figure 1 This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. It should be noted that the steps shown may be executed in a logical order different from that shown in the flowchart. The method includes: Step 110: Obtain the motion input data of the target object; Step 120: Determine the first skeleton pose data of the target object based on the motion input data; Step 130: Map the first skeleton pose data of the target object to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton pose data of the target virtual character; Step 140: Based on the second skeleton pose data, identify the virtual actions to be performed by the target virtual character and obtain multiple candidate virtual actions; Step 150: Based on multiple candidate virtual actions, action control delay information, and the motion constraints of the virtual character corresponding to the target game, determine the target virtual action. The action control delay information is used to indicate the delay between the target object executing the action and the target virtual character responding to the target object's action. Step 160: Control the target virtual character to perform the target virtual action.

[0047] Specifically, in game motion-sensing interaction scenarios, there are inherent differences between the real human skeleton and the virtual character skeleton in terms of bone segment proportions, joint degrees of freedom, and topological structure. Using a fixed mapping method can easily lead to visual anomalies such as joint refraction, clipping, and foot slippage. At the same time, there is a mismatch in granularity between the player's continuous body movements and the discrete skill events in the game. Relying solely on instantaneous angle thresholds for action recognition can easily result in false triggers and missed triggers. Furthermore, the cumulative delay in the motion-sensing control chain from acquisition to rendering leads to poor character responsiveness. Differences in trigger conditions among different player groups affect the fairness of competition, and the various optimization dimensions are isolated from each other and difficult to coordinate and balance.

[0048] Based on this, in the virtual character control method in the game provided in the embodiments of this application, the electronic device can first obtain the motion input data of the target object and calculate the first skeleton posture data that can be translated into real human posture. Then, the first skeleton posture data is mapped and adapted to the second skeleton posture data corresponding to the virtual character skeleton. Subsequently, multiple candidate virtual actions are identified based on the second skeleton posture data. Finally, the target virtual action is determined by combining the motion control delay information and the virtual character motion constraints, and the character is driven to execute. Thus, the real-time performance, stability and fairness of motion interaction can be balanced through multi-dimensional joint decision-making. While ensuring operation and feel, the legality and consistency of the action can also be guaranteed.

[0049] In some embodiments, the target object can be understood as the real player performing motion-sensing operations, which is the source of motion-sensing input data. Specifically, it may include a single player in a single-player game scenario or multiple players in a multiplayer battle scenario.

[0050] In some embodiments, motion input data can be understood as a set of raw data collected by various sensing devices that reflects the body movement state of a target object, and may include single-modal or multi-modal sensing data.

[0051] Exemplary examples show that, in some cases, motion-sensing input data includes one or more of the following: visual skeletal point coordinate data captured by a camera, three-axis acceleration and three-axis angular velocity data output by a wearable inertial measurement unit (IMU), button trigger events and posture data from a game controller, and orientation data captured by auxiliary sensors on the mobile device. Specifically, the camera skeletal point data may contain two-dimensional or three-dimensional coordinates of more than twenty key joints of the human body, the IMU data may be continuously output at a fixed sampling rate, and the controller data is event-triggered discrete data.

[0052] In some embodiments, the first skeleton posture data can be understood as posture data calculated based on somatosensory input data, describing the spatial position and rotation angle of each joint of the target object's real human body, and represented by the human skeleton topology as the organizational form, representing the player's overall body posture at the current moment.

[0053] As an example, when a player makes a punching motion, the electronic device performs time alignment and fusion processing on multiple somatosensory input data. The calculated first skeletal posture data can include posture parameters of multiple joints such as shoulder abduction angle, elbow flexion angle, wrist rotation angle, and trunk twisting angle, as well as the three-dimensional spatial coordinates of each joint point such as head, neck, shoulder, elbow, wrist, hip, knee, and ankle.

[0054] In some embodiments, the target game can be understood as a specific game application using motion-sensing interaction technology. The target game contains game elements such as virtual character art assets, combat rule system, physics engine, and animation system, and is also the carrier environment for virtual character operation and motion-sensing control.

[0055] In some embodiments, the target virtual character can be understood as a game character entity in the target game that is controlled by the player's body movements. It has an independent three-dimensional character model, skeletal binding structure and motion resource library. The player's body movements are ultimately reflected in the behavior changes of the virtual character.

[0056] In some embodiments, the virtual character skeleton can be understood as the skeletal hierarchy structure bound to the target virtual character, which is composed of multiple bone segments and joint nodes organized according to parent-child dependency relationships, and has rotational degree of freedom constraints for each joint and bone segment length definitions.

[0057] Exemplary, the virtual character skeleton may include a hierarchical structure such as root skeleton, spinal skeleton chain, head skeleton, left arm skeleton chain, right arm skeleton chain, left leg skeleton chain, and right leg skeleton chain. Each bone segment has a fixed length attribute, and each joint has a limited range of rotatable angles. Some characters may also include extended skeleton nodes such as weapon attachment bones and equipment attachment bones.

[0058] In some embodiments, the second skeleton posture data can be understood as the posture data obtained by mapping and adapting the first skeleton posture data of a real human body to the virtual character skeleton system. The joint topology, bone segment length ratio and joint rotation range in the second skeleton posture data all conform to the virtual character's own skeletal definition and can be used to drive the virtual character to perform posture presentation and animation playback.

[0059] In some embodiments, a virtual action can be understood as a specific action unit that a virtual character can perform in a game, corresponding to discrete action states such as skill release, normal attack, movement posture, and interactive action in the game. Each virtual action is associated with a corresponding animation clip and game logic triggering conditions.

[0060] Exemplary actions can include various types such as straight punches, hook punches, blocking, jumping, crouching, dodging, and skill activation. Among them, the straight punch action corresponds to an animation clip of the character's arm extending forward with damage assessment; the blocking action corresponds to an animation clip of the character raising their arm in front of their body with damage reduction; and the jumping action corresponds to an animation clip of the character's body moving upward with collision changes.

[0061] In some embodiments, candidate virtual actions can be understood as a set of multiple possible actions that are initially identified based on the current skeleton pose and have a certain matching probability, reflecting the ambiguity caused by pose similarity in the action recognition stage.

[0062] For example, when a player raises their arm to shoulder height and is in motion, the electronic device can identify three candidate virtual actions: a straight punch, a hook punch, and a block. Each candidate corresponds to a different matching probability value, with the straight punch having the highest matching probability and the hook punch and block having relatively lower matching probabilities.

[0063] In some embodiments, motion control latency information can be understood as the time difference and phase difference data between when the target object actually performs a physical action and when the virtual character presents a corresponding response on the display screen, used to characterize the end-to-end latency level of the motion control link.

[0064] For example, motion control latency information may include the sum of time components from various stages such as acquisition latency, processing latency, rendering latency, and display latency, as well as the offset between the player's action phase and the character's animation phase. In one example, the end-to-end time difference corresponding to the motion control latency information in a certain device environment is approximately eighty milliseconds, and the phase difference corresponds to twelve percent of the action cycle.

[0065] In some embodiments, virtual character motion constraints can be understood as a set of restrictions that the target virtual character must meet when performing actions, in order to ensure that the final output action conforms to both visual physics and gameplay design logic, and is an important constraint dimension in the action decision-making stage.

[0066] For example, virtual character motion constraints include two categories: physical constraints and game rule constraints. Physical constraints may include upper limits on joint rotation angles, constraints on constant bone segment length, constraints on foot contact with the ground, and constraints on non-penetration by colliders, which are used to limit the feasibility of virtual character movements in terms of physical performance. Game rule constraints may include restrictions on skill cooldown time, rules on the recovery and stun of actions, requirements on the order of attacks, and rules on damage calculation windows, which are used to limit the virtual character's movements to conform to the gameplay rules of the target game.

[0067] In some embodiments, a target virtual action can be understood as a single virtual action that is finally determined to be executed after multi-dimensional screening and optimization decisions, and can be used to drive a virtual character to perform a corresponding behavior.

[0068] To more clearly illustrate the virtual character control method in the game provided in this application embodiment, please refer to the following exemplary description: First, electronic devices acquire motion input data from the target object through various acquisition terminals such as cameras, inertial sensors, and handheld devices.

[0069] Then, based on the haptic input data acquired by the electronic device, the first skeleton pose data of the target object is determined through processing such as time alignment, multi-source fusion, confidence assessment and pose calculation. The scattered multimodal sensor data is transformed into a unified human skeleton pose representation, thereby obtaining the position and angle information of each joint that can reflect the player's real body posture.

[0070] Next, the electronic device establishes a topological mapping relationship between the real human skeleton and the virtual character skeleton. Combining the bone segment ratio deviation, joint rotation deviation, and kinematic constraints of the virtual character's parent and child bone segments, the first skeleton posture data is transformed and processed through posture transformation solution to obtain the second skeleton posture data of the target virtual character, thereby completing the adaptation and conversion from the real human posture to the game character posture.

[0071] Subsequently, the electronic device performs action intent recognition based on the second skeleton posture data. Combining action timing features and posture change trends, it identifies multiple candidate virtual actions to be performed by the target virtual character, forming a set of action candidate sets with probability distribution.

[0072] Next, the electronic device uses multiple candidate virtual actions as the basis for decision-making, introduces motion control delay information for delay prediction and phase compensation, and combines the motion constraints of the virtual character corresponding to the target game for legality verification. By constructing a unified optimization objective that includes multiple dimensions such as posture stability, reliable intent, controllable delay, legal rules, smooth timing, and group fairness, the optimal target virtual action is determined from the candidate actions, thus achieving comprehensive decision-making under multiple constraints.

[0073] Finally, the electronic device sends the determined target virtual action to the game animation system and logic engine, controls the target virtual character to execute the corresponding action animation and trigger the corresponding game logic, completing a complete control loop from player's sensory input to virtual character's behavioral output.

[0074] Thus, in this embodiment, the motion input data of the target object can be acquired, and based on the motion input data, the first skeleton posture data of the target object can be determined. The first skeleton posture data of the target object is then mapped to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton posture data of the target virtual character. Based on the second skeleton posture data, the virtual action to be performed by the target virtual character is identified, resulting in multiple candidate virtual actions. Based on the multiple candidate virtual actions, action control delay information, and the motion constraints of the virtual character corresponding to the target game, the target virtual action is determined, and the target virtual character is controlled to perform the target virtual action. This allows players and other target objects to complete real-time control of the game virtual character solely through body movements, resulting in a stronger sense of immersion compared to traditional gamepad button control. Furthermore, by adapting and converting the real human posture into the virtual character posture through skeleton mapping, the structural differences between different skeleton systems can be reduced, avoiding joint distortion and clipping anomalies. Meanwhile, by using multi-candidate action recognition and multi-constraint joint decision-making to determine the target action, the delay between the player's action and the virtual character's response, as well as the virtual character's motion constraints, can all participate in the target action decision-making, reducing the probability of misjudgment and avoiding conflicts between independent optimization of each module to a certain extent. Thus, while ensuring the responsiveness of the motion control, it can also ensure the robustness and compliance of the action output.

[0075] In some embodiments provided in this application, the motion-sensing input data includes multiple sub-motion-sensing data, each of which is acquired through a terminal device. Therefore, step 120 includes: performing time alignment processing on the multiple sub-motion-sensing data to obtain multiple processed sub-motion-sensing data; performing fusion processing on the multiple processed sub-motion-sensing data to obtain fused motion-sensing data; and determining the first skeleton posture data based on the fused motion-sensing data.

[0076] Specifically, considering that player motion data in motion-sensing games typically originates from various heterogeneous acquisition devices, each with different sampling frequencies, independent clock bases, and varying timestamp accuracies, and that data transmission may be out of order and jittery, directly sending unaligned multi-channel raw data into the subsequent posture calculation module would cause problems such as misaligned limb movements, abrupt joint position changes, and deviations in skill trigger timing. Furthermore, a single data source lacks robustness to environmental interference; cameras are susceptible to lighting and occlusion; and inertial sensors suffer from cumulative drift, making it difficult to independently provide stable and reliable posture input.

[0077] Based on this, in some embodiments provided in this application, the electronic device can receive sub-motion sensing data from multiple terminal devices. First, time alignment processing is performed on the multiple data streams to map data from different clock domains to the same target time axis and compensate for clock drift, resulting in time-consistent processed sub-motion sensing data. Then, the multiple processed sub-motion sensing data are fused, and the observation results from each data source are combined to generate fused motion sensing data. Finally, the first skeleton pose data of the target object is calculated based on the fused motion sensing data, providing an input basis for subsequent skeleton mapping and motion recognition.

[0078] In some embodiments, sub-sensory data can be understood as a type of raw somatosensory data independently acquired by a single acquisition terminal. It is a component unit constituting a complete somatosensory input, and the data dimensions, sampling frequencies, and physical meanings of different sub-sensory data are different.

[0079] In some examples, sub-motion sensing data may include a sequence of two-dimensional or three-dimensional keypoint coordinates output by a visual camera. This type of data extracts the location of key points of the human body from color or depth images using computer vision algorithms, and has absolute spatial positioning capabilities.

[0080] In some examples, sub-motion data may include acceleration, angular velocity, and magnetometer data output from a wearable inertial measurement unit. This type of data is sampled at a high frequency and is well-suited for capturing high-speed dynamic motion.

[0081] In some examples, sub-motion data may include button events and gesture quaternion data output from a game controller or smart bracelet, used to provide discrete interaction events and local gesture supplementation.

[0082] In some examples, the sub-motion data may include a variety of data from the aforementioned visual camera output, wearable inertial measurement unit output, game controller or smart bracelet output.

[0083] In some embodiments, a terminal device can be understood as an independent hardware terminal with the ability to collect motion signals. Each terminal device operates independently, is equipped with an independent hardware clock, and reports its collected motion data to the main control electronic device.

[0084] In some examples, the terminal device may include visual acquisition devices such as depth cameras and optical motion capture cameras to capture human posture in a non-contact manner.

[0085] In some examples, the terminal device may include wearable devices such as inertial measurement units and wearable wristbands, which acquire kinematic data through sensors that are closely attached to the human body.

[0086] In some examples, the terminal device may include consumer electronic devices with posture sensing capabilities, such as game controllers and smartphones, which can be used for both interactive input and auxiliary posture acquisition.

[0087] In some examples, the terminal device may include a variety of the aforementioned visual acquisition devices, wearable devices, and consumer electronic devices.

[0088] In some embodiments, time alignment processing can be understood as a data preprocessing operation that regularizes multiple heterogeneous time series to a unified time base, used to eliminate clock frequency deviations, initial phase deviations, and accumulated clock drifts between devices, ensuring that all data are strictly synchronized in the time dimension.

[0089] In some examples, time alignment can be achieved through a two-step process: clock domain mapping and drift compensation. Specifically, first, an affine transformation relationship is established from each source clock to a unified clock, converting the original timestamps into a unified clock domain scale. Then, the clock drift of each device is estimated recursively using reference pulses, and the timestamps are dynamically corrected. Finally, the corrected data is resampled onto a unified grid with fixed time intervals, ensuring that each data stream has a corresponding sampled value at every moment.

[0090] In some embodiments, processing sub-sensory data can be understood as sub-sensory data that has undergone time alignment processing. The temporal reference of each processed sub-sensory data is consistent, and the sampling points are aligned in the time dimension, which can be directly used for cross-channel data fusion calculation.

[0091] In some embodiments, fusion processing can be understood as the process of merging multi-channel time-aligned processed sub-sensory data into a single comprehensive data according to a certain weighting strategy.

[0092] In some examples, the fusion process can employ a confidence-weighted mechanism. Specifically, the electronic device evaluates the current quality of each data stream in real time. When the confidence of a data stream decreases due to obstruction, jitter, or signal attenuation, the fusion weight of that data stream is automatically reduced, while the weights of other reliable data sources are increased accordingly, thus preventing anomalies from a single data source from impacting the overall result.

[0093] In some embodiments, the fused somatosensory data can be understood as the output result after fusion processing, which may cover the spatial position and related kinematic information of each joint of the target object at the current moment.

[0094] To more clearly illustrate the processing procedure of multi-source somatosensory data in the embodiments of this application, please refer to the following exemplary description: The electronic device simultaneously connects to multiple terminal devices, each independently acquiring motion-sensing data of the target object and generating corresponding sub-motion-sensing data. Specifically, the visual camera outputs a sequence of human joint coordinates at a sampling rate of 30 frames per second, along with a local camera timestamp; the Inertial Measurement Unit (IMU) outputs acceleration and angular velocity data at a sampling rate of 200 Hz, along with a local IMU timestamp; and the game controller reports button states via event triggering, along with a controller timestamp. Each of the three sub-motion-sensing data streams carries an independent clock reference, with significantly different sampling frequencies. The order in which the data arrives at the main control electronic device may also be out of order due to transmission delays.

[0095] After receiving multiple channels of raw sub-sensory data, the electronic device first performs time alignment processing. The first step involves establishing a mapping relationship between the source clock and the target unified time axis for each data channel. This is achieved by converting the local timestamps of each device into a unified clock domain time scale through frequency scaling and offset shifting, eliminating clock domain differences. The second step involves the electronic device periodically emitting reference calibration pulses. Based on the response time of each device to the reference pulse, the clock drift of each data channel is recursively estimated, dynamically correcting the timestamps and compensating for the slow drift accumulation caused by crystal oscillator differences. The third step involves resampling the compensated data channels onto a unified time grid at fixed intervals through interpolation, ensuring that the three sub-sensory data channels have corresponding sampled values ​​at each target time, and outputting processed sub-sensory data with strictly consistent timing.

[0096] After obtaining the multi-channel processed somatosensory data, the electronic device performs fusion processing. The electronic device evaluates the current observation quality of each channel in real time, calculates the confidence weight of each channel based on dimensions such as joint tracking status, anatomical constraint bias, and multi-sensor observation consistency, and then weights and fuses the processed somatosensory data according to these weights to generate fused somatosensory data that integrates multi-source observation information. When a channel of data is occluded or anomaly-prone, its weight is automatically reduced, while the weights of other reliable data sources are increased accordingly, ensuring the stability of the fusion result.

[0097] Finally, the electronic device determines the first skeleton pose data based on the fused somatosensory data. The electronic device combines the observed position, observation confidence level, and historical trajectory prediction results of each joint point, performs weighted fusion of the observed and predicted values, and introduces an anomaly detection mechanism to remove outliers, outputting the target position of each joint point at each moment, forming structured first skeleton pose data, which serves as the input basis for subsequent skeleton redirection and action intent recognition.

[0098] Thus, in this embodiment, multiple sub-motion sensing data can be time-aligned to obtain multiple processed sub-motion sensing data, and these multiple processed sub-motion sensing data can be fused to obtain fused motion sensing data. Based on the fused motion sensing data, the first skeleton posture data is determined. This time alignment addresses timing discrepancies between output data from different terminal devices, ensuring phase synchronization of all data streams and avoiding limb distortion and motion judgment errors caused by time misalignment. Simultaneously, multi-source data fusion leverages the complementary advantages of different data acquisition devices, thereby improving the stability and anti-interference capability of posture estimation in complex environments.

[0099] In some embodiments provided in this application, the step of performing time alignment processing on multiple sub-sensory data to obtain multiple processed sub-sensory data includes: mapping each sub-sensory data to a target time axis to obtain mapped sub-sensory data corresponding to each sub-sensory data; determining the clock drift amount of each sub-sensory data relative to the target time axis based on the timestamp information of each sub-sensory data and the target time axis; and performing compensation processing on the mapped sub-sensory data corresponding to each sub-sensory data based on the clock drift amount corresponding to each sub-sensory data to obtain multiple processed sub-sensory data.

[0100] Specifically, each motion-sensing data acquisition device (i.e., terminal device) may be equipped with an independent hardware clock crystal oscillator. Due to differences in manufacturing processes, operating temperature fluctuations, and power supply voltage variations, the clock frequencies of different devices exhibit slight but not negligible deviations. Furthermore, the startup times of the devices differ, resulting in differences in the timestamp references of the various sub-motion-sensing data. If only a one-time initial time mapping is performed without tracking clock drift, the cumulative deviation of the clocks of each device will continue to increase over time, causing problems such as motion phase misalignment and joint data asynchrony. Moreover, in high-speed motion scenarios, millisecond-level timing deviations can lead to significant distortion in the posture calculation of movements such as punching and turning, directly affecting the accuracy and smoothness of motion-sensing tracking.

[0101] Based on this, in some embodiments provided in this application, the electronic device first maps each sub-sensory data to a unified target time axis through clock domain transformation, eliminating the initial clock offset and frequency difference between devices to obtain mapped sub-sensory data; then, it continuously compares the timestamp information of each sub-sensory data with the reference time of the target time axis to calculate the real-time clock drift of each data relative to the target time axis. Finally, the electronic device performs timestamp compensation correction on each mapped sub-sensory data according to the calculated drift amount to offset the cumulative deviation caused by clock drift, and finally outputs processed sub-sensory data with precise timing alignment.

[0102] In some embodiments, the target timeline can be understood as a unified time reference set by the system, or as a reference standard for aligning each sub-sensory data, providing a unique temporal reference system for the entire somatosensory processing chain. The target timeline typically has a fixed and uniform time scale, and the data from all acquisition devices must eventually be normalized to this timeline to ensure the temporal consistency of subsequent fusion and calculation.

[0103] In some examples, the target timeline can be constructed based on the system clock of the main control device or on an independent high-precision clock source. The time interval of the target timeline can be set to the millisecond or even sub-millisecond level according to the real-time requirements of the motion sensing system, thereby ensuring that the time alignment accuracy meets the requirements of high-speed motion capture.

[0104] In some embodiments, the mapped sub-motion data can be understood as the sub-motion data after initial clock domain mapping, that is, the intermediate result obtained by converting the local timestamps of each device onto the target time axis through clock domain transformation. The mapped sub-motion data completes coarse-grained time alignment, but due to the continuous drift of device clocks, a single static mapping is insufficient to maintain long-term synchronization accuracy, so further drift compensation is still required.

[0105] In some examples, clock domain mapping can be implemented using affine transformations. By using two parameters—a frequency scaling factor and a time offset—local clocks with different scales and starting points are uniformly converted to the target time axis's time scale. The frequency scaling factor corrects for differences in device clock speed, while the time offset corrects for phase differences caused by different device startup times.

[0106] In some embodiments, the timestamp information can be understood as the acquisition time marker carried by each frame of sub-sensory data, which is marked by the local clock time when the data is generated by the acquisition device to identify the sampling time point of the data frame.

[0107] In some examples, the timestamp information can be an absolute time value generated by the device's hardware clock, or a relative time value from when the device starts counting after startup. By comparing the differences in the timestamps of the same reference event recorded on different devices, the degree of deviation of each device's clock relative to the target time axis can be calculated.

[0108] In some embodiments, clock drift can be understood as the cumulative deviation of the local clock of each acquisition device relative to the target time axis, which can reflect the degree of accumulation of time error caused by factors such as inaccurate crystal frequency and temperature drift of the device clock over time.

[0109] In some examples, clock drift can be recursively estimated using the reference pulse method. Specifically, electronic devices periodically emit a uniform reference calibration pulse, and each device records its local timestamp when it receives the reference pulse. The local timestamp is compared with the standard time of the reference pulse on the target time axis to obtain the drift observation value at the current time. Then, the historical drift estimates and the new observation value are weighted and updated using a smoothing filter to obtain a stable real-time clock drift.

[0110] In some embodiments, the compensation process can be understood as an operation of correcting and adjusting the timestamp of the mapped sub-sensory data according to the calculated clock drift amount, in order to offset the error caused by the device clock deviating from the target time axis, so that the corrected timestamp matches the standard time of the target time axis.

[0111] In some examples, compensation processing is performed independently per channel. Each sub-sensory data is corrected using its own corresponding drift amount. Specifically, the timestamp of each frame of mapped sub-sensory data is added to or subtracted from the clock drift amount of the corresponding channel. After compensation, the data can be further resampled onto a unified time grid through interpolation to obtain processed sub-sensory data with fully aligned timing.

[0112] To more clearly illustrate the specific process of time alignment in the embodiments of this application, please refer to the following exemplary description: The electronic device simultaneously connects to three data acquisition devices: a visual camera, an IMU (Integrated Device Unit), and a game controller. Each device generates sub-motion sensing data with a local timestamp. The camera samples at 30 frames per second, with a nominal crystal oscillator frequency of 30 MHz. The IMU samples at 200 Hz, with a nominal crystal oscillator frequency of 16 MHz. The game controller uses event-triggered sampling, with a nominal crystal oscillator frequency of 8 MHz. The actual crystal oscillator frequencies of the three devices all deviate, and their startup times differ, resulting in completely inconsistent local timestamp references.

[0113] First, the electronic device performs an initial mapping step, which establishes an affine mapping relationship from the local clock to the target time axis for each sub-motion data stream. Specifically, for camera data, its local time scale is converted to the target time axis scale using a frequency scaling factor, and an initial offset is added to correct the start-up time difference, resulting in the mapped sub-motion data corresponding to the camera. Similarly, mapping transformations are performed on IMU data and controller data respectively, resulting in three mapped sub-motion data streams. After this step, the time base of the three data streams is roughly unified.

[0114] Subsequently, the electronic equipment continuously performs the drift estimation step. Specifically, the electronic equipment sends a reference calibration pulse at fixed time intervals, and all three devices record the local timestamp when the pulse is received. The electronic equipment compares the local timestamps reported by each device with the standard time of the reference pulse on the target time axis, and calculates the current drift observation value for each of the three data streams. Then, a first-order recursive filtering method is used to merge the historical drift estimates with the new observation values ​​according to weights, updating the real-time clock drift of each data stream. This estimation process is repeated cyclically, which can dynamically track the slow drift of the device clock caused by factors such as temperature changes and voltage fluctuations.

[0115] Finally, the electronic device performs a compensation process. Specifically, for the mapped sub-motion data from the camera channel, the timestamp of each frame is corrected using the real-time clock drift corresponding to the camera. For the mapped sub-motion data from the IMU channel and the controller channel, corrections are performed independently using their respective drift values. After correction, the timestamps of all three data streams are precisely aligned to the target timeline, resulting in processed sub-motion data. The electronic device can then resample the compensated data streams to a unified time grid at fixed intervals via interpolation, preparing for multi-source data fusion.

[0116] Thus, in this embodiment, each sub-motion sensing data can be mapped to a target timeline to obtain mapped sub-motion sensing data corresponding to each sub-motion sensing data. Based on the timestamp information of each sub-motion sensing data and the target timeline, the clock drift of each sub-motion sensing data relative to the target timeline is determined. Based on the clock drift of each sub-motion sensing data, the mapped sub-motion sensing data corresponding to each sub-motion sensing data is compensated separately, resulting in multiple processed sub-motion sensing data. This eliminates clock domain differences between devices by mapping each sub-motion sensing data to the target timeline, ensuring the timing consistency of multi-source motion sensing data. Simultaneously, compensation based on the determined clock drift ensures that each data stream can be corrected to the target timeline, further improving the timing consistency of multi-source motion sensing data and avoiding situations such as asynchronous upper and lower body movements, skill trigger timing deviations, and joint position jumps caused by time misalignment.

[0117] Furthermore, to more clearly illustrate the process of determining the fused somatosensory data in the embodiments of this application, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. Also, please... Figure 2 Based on the following exemplary description, that is: It is understandable that different acquisition devices have independent clocks, and the arrival time of data and the sampling interval are uncertain. Therefore, if timeline alignment is not completed in advance, all subsequent calculations will be based on data with inconsistent timing. In other words, if there is a millisecond-level deviation in the timelines of multiple data streams, problems such as limb timing misalignment and deviations in motion orientation calculation will occur in the skeleton mapping and action recognition stages. Among these, samples with misaligned timing will directly affect the model training effect, leading to a decrease in model convergence performance and recognition accuracy.

[0118] Based on this, the embodiments of this application can firstly utilize the frequency scaling factor. a s and fixed time offset b s Eliminate the inherent deviation between clock domains of different devices; then use clock drift compensation. d s,k Online estimation and correction of clock drift are achieved; finally, the data from each channel are mapped to a time grid with fixed intervals to obtain a fused sequence. X ( t u Understandably, this output maintains consistency in action phase while relying on the jitter variance index. Quantifying the input timing quality provides a basis for downstream modules to dynamically adjust the confidence threshold.

[0119] Specifically, assume that the electronic device is simultaneously connected to three terminal devices: a camera, an IMU, and a handset, and that the original observation data (i.e., sub-input data) from the camera, IMU, and handset are as follows: C ={ c n , }、 I ={ i m , }、 G ={ g q , }, define a unified time base as t .

[0120] First, as shown in formula (1), timestamp standardization is performed on each original observation data, that is, an affine mapping relationship is established from the source clock to the unified clock, and the conversion of the time scale of each device to the common time reference is completed.

[0121] (1) In the formula, t Indicates a unified time base; () indicates the first s Clock affine mapping function for the data source; tIndicates the local time of the source device; a s Indicates the frequency scaling factor; indicates the difference in the rate of fixed time offset between the device clock and the standard clock; b s Indicates a fixed time offset; indicates the difference between the device startup time and the system reference time; s Indicates the data source type identifier; c This represents the raw observation data from the camera data source; i This indicates that the IMU receives raw observation data from an inertial data source; g This represents the raw observation data from the handle data source.

[0122] Then, since a single calibration cannot accommodate the clock drift that occurs over time, a reference pulse is introduced from the periodic output of the electronic device to continuously estimate and compensate for clock drift. r k Drift estimation compensation is performed using the recursive estimation algorithm for drift amount shown in formula (2).

[0123] (2) It is understandable that formula (2) can calculate the compensated timestamp based on the original observation timestamp. In formula (2), d s,k Indicates the first s Road data source number k The estimated clock drift compensation amount; r This represents the smoothing coefficient, which controls the mixing ratio of historical estimates and new observations; r k Indicates the first k One reference pulse timestamp; t s,k Indicates the first s Road data source number k The original timestamp of each observation; s,k This represents the corrected timestamp after drift compensation. It's worth noting that the parameter... r This is used to adjust the smoothing characteristics of the filter. The larger the value, the more the algorithm relies on historical estimation results; the smaller the value, the more sensitive the algorithm is to the latest observation data.

[0124] Next, each sequence is resampled to a uniform time grid. r u = u Δ t Then, as shown in formula (3), confidence weights are used to achieve multi-channel signal fusion in order to realize event time mapping and thus generate a unified output sequence.

[0125] X ( t u ) =S s∈{c,i,g} oh s ( t u ) ·R s ( s ,t u ) , Σ s oh s ( t u ) = 1(3) In the formula, X ( t u ) represents a unified grid t u The fused output signal at each moment; oh s ( t u )express t u Time of the first s Confidence weights of the road signal; R s (·) indicates the first s Interpolation resampling operator for the signal path; s Indicates the first s The signal sequence after time compensation is passed through the path; t u The first unified time grid represents the u This is a specific point in time. It is worth noting that... R s (·) is used to interpolate and map data acquired at non-uniform time intervals to a uniform time grid.

[0126] In addition, to quantify the timing alignment effect, a window jitter variance constraint condition is constructed as shown in formula (4).

[0127] (4) In the formula, σ jit ² represents window jitter variance, quantifying time alignment quality; U Indicates the total number of time points within the window; D u Indicates the first u The interval between adjacent correction timestamps; Indicates the average time interval within the window; ejit This represents the tolerance threshold for jitter variance. It's worth noting that when the jitter variance σ... jit ²When the threshold is exceeded, the electronic device automatically reduces the weight corresponding to the high-jitter acquisition mode. oh s This ensures that subsequent modules obtain input data with stable timing.

[0128] In some embodiments provided in this application, the fused somatosensory data includes the observed position of each joint of the target object at the current moment. Therefore, the step of determining the first skeleton pose data based on the fused somatosensory data includes: obtaining the confidence level and predicted position of the observed position of each joint at the current moment; determining the target position of each joint at the current moment based on the observed position, the confidence level, and the predicted position; and determining the target position of each joint at the current moment as the first skeleton pose data. The predicted position is determined by the observed position at least one moment prior to the current moment.

[0129] Specifically, considering that the joint observation data obtained from multi-source fusion is still affected by factors such as acquisition noise, momentary occlusion, and sensor jitter, the observation position in a single frame may fluctuate randomly or even jump abnormally. If directly used as the skeleton posture output, it will lead to visual defects such as joint tremors and bone shaking. At the same time, local occlusion and signal loss may occur during the motion sensing acquisition process, which may lead to skeleton breakage or limb drift. In addition, human movement has inherent continuity and inertia. Simply relying on instantaneous observation while ignoring temporal correlation cannot effectively filter noise, nor can it reasonably compensate for observation failures.

[0130] Based on this, in some embodiments provided in this application, the electronic device can obtain the observed positions of each joint from the fused haptic data, and simultaneously calculate the predicted position at the current moment based on the observed data from historical moments, assigning a corresponding confidence level to each observed position. Subsequently, based on the observed position, the confidence level of the observed position, and the predicted position, the target position of each joint is determined through weighted fusion, with higher confidence levels having greater weight, and lower confidence levels relying more on the predicted value for a smooth transition. Finally, the target positions of all joints are combined into the first skeleton pose data and output to the subsequent skeleton mapping module.

[0131] In some embodiments, a joint can be understood as a key node on the human skeleton that has kinematic significance. It is the basic unit that constitutes the skeletal posture. Each joint forms a hierarchical connection relationship according to the human anatomical topology, and together they describe the complete human posture.

[0132] In some examples, joints may include major joints of the limbs such as the shoulder, elbow, wrist, hip, knee, and ankle joints, as well as spinal joints such as the cervical, thoracic, and lumbar vertebrae, and key rigid body nodes such as the head and pelvis. They may also include fine peripheral joints such as finger and toe joints. The specific number and distribution depend on the accuracy level of the motion sensing acquisition scheme and the application scenario requirements.

[0133] In some embodiments, the observed position can be understood as a direct measurement result reflecting the spatial position of the joint at the current moment, obtained by multi-source somatosensory data fusion calculation. It is the direct observation output of the sensor on the real joint position and is subject to a certain degree of random noise due to the influence of acquisition accuracy and environmental interference.

[0134] In some examples, the observation position can be represented as Cartesian coordinates in three-dimensional space, obtained by fusing skeletal point detection from a visual depth camera with kinematic data from an inertial measurement unit.

[0135] In some examples, the observation position can be represented as pixel coordinates in a two-dimensional image plane, directly output by the human pose estimation algorithm of a monocular camera.

[0136] In some examples, the observation position can be represented as a relative displacement with respect to the root node, used for hierarchical skeletal kinematics calculations.

[0137] In some embodiments, the confidence level of the observation position can be understood as a quantitative indicator that measures the credibility of the observation position of the current frame joint. The higher the confidence level, the more reliable and trustworthy the observation result is. The lower the confidence level, the greater the impact of occlusion, noise or sensor failure on the observation, and the lower its reference value.

[0138] In some examples, the confidence level of the observation location can be expressed as a normalized value in the range of 0 to 1, calculated by combining multiple dimensions of factors such as the visual tracking status of the joint, the degree of deviation between the observation results and anatomical constraints, and the consistency of observations among multiple sensors. The confidence level approaches 0 when the joint is completely occluded, and approaches 1 when the joint is clearly visible and the multi-source observations are consistent.

[0139] In some embodiments, the predicted position can be understood as the expected position of the key point at the current moment calculated by a kinematic model or time-series prediction algorithm based on historical observations or historical target positions at least one frame prior to the current moment. This reflects the continuity and inertial laws of human movement and is used to provide a smooth position reference when the observation quality deteriorates.

[0140] In some examples, the predicted position can be extrapolated from the position and velocity of the previous frame using a uniform motion model, suitable for regular motion scenarios. Alternatively, the predicted position can be recursively estimated using a Kalman filter algorithm combined with the state equations and observation equations of the electronic device, offering noise suppression capabilities. Or, the predicted position can be predicted using a higher-order motion model, such as an acceleration model, adaptable to high-speed, variable-acceleration motion scenarios.

[0141] In some embodiments, the target location can be understood as the final joint location determined after the fusion of observation and prediction. It is the optimal estimate of the actual joint location, combining the spatial accuracy of observation with the temporal smoothness of prediction.

[0142] To more clearly illustrate the process of determining the skeleton pose in the embodiments of this application, please refer to the following exemplary description: The electronic device extracts the observed positions of all joints of the target object at the current moment from the fused somatosensory data. Each observed position corresponds to a three-dimensional spatial coordinate, representing the joint's spatial position measured by the sensor fusion. Simultaneously, the electronic device reads the historical motion state of each joint at previous moments, including position and velocity information, and extrapolates the predicted position of each joint at the current moment based on a uniform motion model. For joints with drastic changes in motion state, the electronic device can also employ a Kalman filter framework, combining the state transition matrix and process noise covariance for recursive estimation to obtain prediction results that better conform to the laws of motion physics.

[0143] The electronic device synchronously calculates the observation position confidence score for each joint at its current observation position. The confidence score assessment integrates multiple dimensions: first, the visual tracking status of the joint; if the visual algorithm marks the joint as occluded or lost, the corresponding confidence score component decreases; second, the degree of deviation between the observation position and human anatomical constraints; if the observation result exceeds the normal range of motion of the joint, the corresponding confidence score component decreases; and third, the consistency of observations among multiple sensors; if there is a significant deviation between visual data and inertial data regarding the motion trend of the joint, the corresponding confidence score component decreases. These three components are weighted, fused, and normalized to a final confidence score value in the range of 0 to 1.

[0144] After obtaining the observed position, the confidence level of the observed position, and the predicted position, the electronic device independently solves for the target position at each joint. The electronic device uses the confidence level of the observed position as a weighting criterion to perform a weighted fusion of the observed and predicted positions. When the confidence level is high, the weight of the observed position is dominant, and the target position closely approximates the measured value, ensuring the accuracy of attitude following. As the confidence level gradually decreases, the weight of the predicted position increases accordingly, and the target position smoothly transitions towards the predicted value. When the confidence level is extremely low or even when the observation is completely ineffective, the target position is primarily dominated by the predicted position, ensuring the continuity of joint movement.

[0145] After the target positions of all joints in the body are calculated, the electronic equipment organizes the target positions of all joints in an orderly manner according to the topological structure of the human skeleton, forming complete first skeleton posture data.

[0146] Thus, in this embodiment, the confidence level and predicted position of each joint at the current moment can be obtained. Based on the observed position, confidence level, and predicted position of each joint, the target position of each joint at the current moment is determined, and the target position of each joint at the current moment is defined as the first skeleton posture data. This achieves a balance between observation accuracy and motion smoothness through a confidence-weighted mechanism. This prioritizes posture accuracy when observation is reliable, and automatically switches to a prediction-driven smooth transition when observation deteriorates, suppressing joint tremors, position jumps, and instantaneous shifts. Furthermore, by introducing predicted positions, the temporal inertia of human motion is fully utilized, providing a fallback capability for common degradation scenarios such as local occlusion and instantaneous signal loss. This ensures that the skeleton posture continues to evolve even when observation is interrupted, avoiding bone fractures and limb drift.

[0147] In some embodiments provided in this application, the confidence level of the observation location is determined in the following ways: Based on the tracking status of the target object's joints, determine the first confidence component corresponding to the joint; based on the deviation between the observed position of the joint and the anatomical constraints of the joint, determine the second confidence component corresponding to the joint; based on the observation consistency among multiple sensors corresponding to the joint, determine the third confidence component corresponding to the joint; based on at least one of the first confidence component, the second confidence component, and the third confidence component, determine the confidence of the observation position.

[0148] Specifically, considering that the quality of observation data at each joint fluctuates in real time with environmental conditions, movement amplitude, and occlusion during the fusion of multi-source motion sensing data to generate skeleton pose, assigning fixed weights to all observation data would lead to a significant deviation of the fusion result from the true pose when data quality deteriorates. For example, when a player turns to the side, causing limb occlusion, the error in joint position detected by vision increases sharply. If the fusion is still performed with normal weights, it will cause skeleton clipping or abnormal jitter. On the other hand, uniformly adopting a conservative low-weight strategy would waste the accuracy potential of high-quality observation data.

[0149] Based on this, in some embodiments provided in this application, the electronic device can calculate confidence components from three dimensions: First, a first confidence component is calculated based on the joint tracking status to assess the degree to which the target is continuously and stably tracked. Second, a second confidence component is calculated based on the deviation between the observed position and anatomical constraints to assess whether the observation results conform to the limitations of human physiological structure. Third, a third confidence component is calculated based on the observation consistency among multiple sensors to assess whether the judgments of different data sources on the same joint position are consistent. Finally, the confidence of the observed position of the joint is determined by combining at least one of the three confidence components, providing a quantitative credibility index for weight allocation in the subsequent fusion process.

[0150] In some embodiments, the observation location confidence score can be understood as a numerical index used to quantify the credibility of the observation location of a certain key point of the target object at the current time. The value is usually between 0 and 1. The higher the value, the more reliable the observation result is, and the greater the weight it occupies in the subsequent fusion calculation.

[0151] In some embodiments, the tracking status can be understood as an indicator of the operating status of the motion sensing electronic device continuously tracking the target joint, which reflects the basic existence and continuity of the joint detection signal, and typically includes multiple status levels such as normal tracking, partial occlusion, and tracking loss.

[0152] In some examples, the tracking status is marked as normal tracking when the player's arm is fully within the camera's field of view and the joint features are clearly identifiable. When the player turns to the side, causing the elbow joint to be partially obscured by the torso, the tracking status is downgraded to partial obscuration. When the player turns around, causing the shoulder joint to completely move out of the detection range, the tracking status is marked as lost tracking.

[0153] In some embodiments, the first confidence component can be understood as a confidence component calculated based on the key point tracking state, directly reflecting the detection effectiveness at the signal level. The more stable the tracking state and the clearer the detection features, the higher the value of the first confidence component. Conversely, the lower the value, the more unstable the tracking state and the clearer the detection features.

[0154] In some embodiments, the observed position of a joint can be understood as the position coordinates of the target object joint in the spatial coordinate system obtained directly by a motion sensor.

[0155] In some embodiments, anatomical constraints can be understood as the limitations on the range of motion of joints and the connection relationship of bones formed based on the physiological structure of the human body. These are physical boundary conditions that the human body must follow for movement, including but not limited to the maximum extension angle of the joint, the limit of the bending direction, the boundary of the rotational degree of freedom, and the proportional relationship of the length of adjacent bone segments.

[0156] In some examples, the backward extension angle of the human elbow joint is usually no more than 10 degrees, the knee joint cannot bend to the side, and the shoulder joint, although having a large range of motion, also has a rotational limit. These are all typical examples of anatomical constraints.

[0157] In some embodiments, deviation can be understood as the difference between the kinematic parameters derived from the actual observed position of the joint and the anatomical constraint limits, used to measure the degree to which the observation results conform to human physiological structure. The smaller the deviation, the more reasonable the observation results; the larger the deviation, the higher the probability of abnormality in the observation results.

[0158] In some embodiments, the second confidence component can be understood as a confidence component calculated based on the deviation between the observation location and anatomical constraints, used to verify the credibility of the observation data from the perspective of structural rationality, and capable of filtering out abnormal detection values ​​that violate the laws of human movement.

[0159] In some embodiments, multiple sensors can be understood as different types of detection devices used to collect haptic input data. Each sensor operates independently based on different detection principles and can provide observation results of joint positions, providing a multi-source data foundation for cross-validation. In some examples, multiple sensors may include a vision camera, an inertial measurement unit, a handle posture sensor, etc., where the camera infers joint positions through image recognition, the inertial measurement unit infers position changes through acceleration and angular velocity integration, and the handle provides precise posture information of the hand nodes through a built-in posture sensor.

[0160] In some embodiments, observation consistency can be understood as the degree of agreement between the observations obtained by multiple sensors for the same key point in terms of spatial location and motion trend. The closer the outputs of each sensor are and the more synchronized their changing trends, the higher the observation consistency. The more significant the differences and the more contradictory the trends, the lower the observation consistency.

[0161] In some embodiments, the third confidence component can be understood as a confidence component calculated based on the consistency of multi-sensor observations, which verifies the reliability of the observation results through cross-comparison of multi-source data. When the detection results of different sensors corroborate each other, it indicates that the observation reliability is high. When the detection results show significant discrepancies, it indicates that at least one sensor has a bias, and the confidence level needs to be reduced accordingly.

[0162] To more clearly illustrate the process of determining the confidence level of the observation location in the embodiments of this application, please refer to the following exemplary description: First, the first confidence component is calculated. The electronic device monitors the tracking status of each joint in real time to determine whether the joint is within the effective tracking range. If the joint appears completely within the field of view of all detection sensors, the feature points are clear, and stable detection occurs continuously for multiple frames, the tracking status is normal, and the first confidence component takes a high value close to 1. If the joint is partially occluded and can only be detected by some sensors, or if the detected features are blurry or jittery, the tracking status is downgraded, and the first confidence component decreases accordingly to a medium range of 0.3 to 0.7, depending on the degree of occlusion. If the joint completely moves out of the detection range or is completely occluded, and the electronic device cannot obtain effective observations, the tracking status is marked as lost, and the first confidence component drops to a low value of 0 or close to 0.

[0163] Next, the second confidence component is calculated. The electronic device substitutes the observed position of the current joint point into the complete skeletal topology, and combines the positional information of adjacent joints to calculate the actual bending angle, rotation angle, and relative proportions of bone segments, as well as other kinematic parameters. These parameters are then compared with pre-stored human anatomical constraint parameters to calculate the deviation value. If the joint angles and bone segment proportions derived from the observations are within the normal physiological range and the deviation is minimal, the second confidence component is set to a high value. If the observation results are close to but not exceeding the anatomical limits, the deviation is moderate, and the second confidence component is adjusted downwards accordingly. If the observation results significantly exceed the physiological limits, such as the appearance of unreasonable phenomena like elbow reflexion or knee lateral flexion, the deviation increases significantly, and the second confidence component decreases substantially.

[0164] Next, the third confidence component is calculated. The electronic device extracts independent observations of the same joint point from different sensor data sources, calculates the spatial distance difference between the position coordinates obtained from each sensor, and the trend differences in motion speed and direction, and comprehensively evaluates the observation consistency. If the joint positions output by multiple sensors such as the camera, inertial measurement unit, and handle are highly consistent and the motion trends are synchronized, the observation consistency is high, and the third confidence component is set to a high value. If there are minor deviations within an acceptable range between some sensors, the observation consistency is moderate, and the third confidence component is moderately reduced. If the observation results from different sensors differ significantly, or even if there are contradictory positions or opposite motion trends, the observation consistency is low, and the third confidence component is significantly reduced.

[0165] Finally, the confidence level of the observation position is determined by combining all components. The electronic device can choose to use all three components or at least one of them for fusion calculation, depending on the actual application scenario. The fusion method can employ various strategies such as weighted summation, product combination, or minimum value. For example, in conventional motion-sensing scenarios, different weights can be assigned to the three components, and then a weighted sum can be obtained to obtain the final confidence level. In competitive scenarios with high requirements for structural rationality, the weight of the second confidence level component can be increased. The final confidence level of the observation position is a continuous value between 0 and 1, used to guide the weight allocation of observed and predicted data during subsequent joint position fusion.

[0166] Thus, in this embodiment, a first confidence component can be determined based on the tracking status of the target object's joints, a second confidence component can be determined based on the deviation between the joint observation position and the anatomical constraints, a third confidence component can be determined based on the observation consistency among multiple sensors, and the observation position confidence can be determined based on at least one confidence component. This allows for a fine quantitative evaluation of the observation quality of each joint, avoiding outliers from directly interfering with skeletal posture calculation and effectively reducing visual anomalies such as limb jumps, joint reversals, and position shifts in virtual characters.

[0167] In some embodiments provided in this application, the step of determining the target position of each joint at the current moment based on the observed position, the confidence level of the observed position, and the predicted position of each joint includes: determining a joint position anomaly score based on the deviation between the observed position and the predicted position of each joint; if the joint position anomaly score is less than or equal to a preset score threshold, determining the observed position weight and the predicted position weight corresponding to each joint based on the confidence level of the observed position of each joint; and performing weighted processing on the observed position and the corresponding observed position weight of each joint, and the predicted position and the corresponding predicted position weight of each joint to obtain the target position of each joint at the current moment.

[0168] Specifically, considering that joint observation positions are inevitably affected by factors such as sensor noise, limb occlusion, lighting changes, and false detections by detection algorithms during real-time motion posture estimation, random fluctuations and instantaneous errors are unavoidable. If the original observation position is directly used as the skeleton posture output, it will cause joints to jitter frequently in space and experience instantaneous jumps. In severe cases, it will produce abnormal positions that do not conform to human anatomy, causing visual problems such as character skeleton distortion and limb clipping. On the other hand, although the position predicted by simply relying on historical trajectories is smooth and stable, it will introduce response delays and will not be able to keep up with the player's real fast movements in time, resulting in a decrease in motion tracking and sluggish operation.

[0169] Based on this, in some embodiments provided in this application, the electronic device may employ an anomaly detection-driven adaptive weighted fusion scheme. Specifically, firstly, the deviation between the observed and predicted joint positions at the current moment is calculated to generate a joint position anomaly score, quantifying the degree of anomaly in the current observation. Then, the anomaly score is compared with a preset score threshold. If the anomaly score does not exceed the threshold, it indicates that the current observation is within a reasonable fluctuation range. Then, the weights of the observed and predicted positions are dynamically determined based on the confidence level of the observed joint position. When the confidence level is high, the observed weight is increased to ensure tracking accuracy; when the confidence level is low, the predicted weight is increased to improve smoothness. Finally, the observed and predicted positions are multiplied by their corresponding weights and summed to obtain the target position of the joint at the current moment, achieving a dynamic balance between smoothness and real-time performance.

[0170] In some embodiments, the joint position anomaly score can be understood as a numerical index used to quantify the degree of deviation between the current observed position and the expected motion, reflecting the degree of anomaly in the current joint observation results. This score is calculated based on the deviation between the observed position and the predicted position; the greater the deviation, the higher the anomaly score, indicating that the current observation results are less consistent with historical motion trends and that there is a greater possibility of noise or false detection.

[0171] In some examples, deviation calculation can use Euclidean distance, which directly calculates the straight-line distance between the observed and predicted positions in three-dimensional space. Mahalanobis distance can also be used, normalizing the deviations by considering the differences in motion variance across dimensions. Weighted distance can also be used, assigning different weights to deviations in different directions based on joint kinematic constraints.

[0172] In some embodiments, the preset score threshold can be understood as a pre-defined anomaly determination boundary value, used to distinguish between normal movement fluctuations and abnormal observation jumps. When the joint position anomaly score is less than or equal to the threshold, the current observation is determined to be within an acceptable range of normal fluctuations, and observation and prediction information can be integrated through weighted fusion. When the anomaly score exceeds the threshold, the observation result is determined to be significantly abnormal, and other processing strategies need to be adopted.

[0173] In some examples, the preset score threshold needs to be set comprehensively based on the application scenario, sensor accuracy, and motion type. For high-precision motion capture equipment scenarios, the threshold can be set relatively low, making it more sensitive to anomaly detection. For ordinary consumer-grade camera scenarios, the threshold can be appropriately relaxed to avoid misjudging normal detection fluctuations as anomalies.

[0174] In some embodiments, the observation location weight can be understood as the weight coefficient assigned to the current observation location during the weighted fusion process, representing the proportion of the observation result's contribution to the final target location calculation. The observation location weight is positively correlated with the observation location confidence level; the higher the confidence level, the greater the observation location weight.

[0175] In some examples, when the joint is unobstructed, multi-sensor observations are consistent, and anatomical constraints are met, the confidence level of the observation position is high, and the weight of the observation position can be set to a larger value. The electronic device trusts the current real-time observation results more, ensuring the accuracy of motion tracking. When the joint is partially obstructed or there is some noise in the observation, the confidence level decreases, and the weight of the observation position is reduced accordingly to reduce the impact of unreliable observations on the results.

[0176] In some embodiments, the predicted location weight can be understood as the weight coefficient assigned to the predicted location during the weighted fusion process, representing the proportion of contribution of historical movement trend prediction to the final target location calculation. The predicted location weight is negatively correlated with the confidence level of the observed location and is typically equal to the sum of the observed location weights.

[0177] In some examples, when observation confidence decreases, the predicted position weight automatically increases, and electronic devices rely more on the smooth continuation of historical motion trajectories to avoid position jumps caused by abnormal observations. When observation confidence recovers, the predicted position weight decreases accordingly, and real-time observation becomes dominant.

[0178] In some embodiments, weighting can be understood as the process of multiplying the observed position with its weight, multiplying the predicted position with its weight, and then adding the two together to obtain the fused target position.

[0179] To more clearly illustrate the process of determining the joint target position in the embodiments of this application, please refer to the following exemplary description: The electronic device performs position calculations for each joint in each frame. First, it acquires the current observed position of the joint, which is derived from multi-source fused motion observation data. Simultaneously, based on the target position of the joint at a historical moment, the electronic device uses kinematic extrapolation to obtain the predicted position for the current moment.

[0180] The electronic device then calculates the spatial deviation between the observed and predicted positions and converts this deviation into a joint position anomaly score. For example, for the right wrist joint, if the observed and predicted positions differ by two centimeters, the calculated anomaly score is low, considering the normal range of motion. However, if the observed position suddenly shifts by thirty centimeters, far exceeding the normal range of motion, the anomaly score increases significantly.

[0181] The electronic device compares the calculated joint position anomaly score with a preset score threshold. If the anomaly score is less than or equal to the preset threshold, it determines that the current observation is within the normal fluctuation range and reads the confidence level of the current observation position of the joint.

[0182] Based on the confidence level of the observed location, the electronic equipment dynamically determines the weights of the observed and predicted locations. For example, when the confidence level is 0.9, the weight of the observed location can be set to 0.85, and the weight of the predicted location can be set to 0.15. When the confidence level drops to 0.4, the weight of the observed location is adjusted to 0.4, and the weight of the predicted location is adjusted to 0.6.

[0183] Finally, the electronic device performs weighted processing, multiplying the observed position by the observed position weight and the predicted position by the predicted position weight, and then summing the results to obtain the target position of the joint at the current moment.

[0184] Thus, in this embodiment, a joint position anomaly score can be determined based on the deviation between the observed and predicted positions of each joint point. If the joint position anomaly score is less than or equal to a preset score threshold, the observed position weight and predicted position weight corresponding to each joint point are determined based on the confidence level of the observed position. Furthermore, the observed position and its corresponding weight, as well as the predicted position and its corresponding weight, are weighted to obtain the target position of each joint point at the current moment. This allows for the differentiation between normal motion fluctuations and abnormal observation jumps by comparing the joint position anomaly score with the preset score threshold, avoiding the misinterpretation of sensor noise or false detections as real motion. Moreover, by dynamically adjusting the weights based on the confidence level of the observed position, an adaptive adjustment of the fusion strategy is achieved. When the quality of the observed data is good, observation is prioritized to ensure the accuracy and real-time performance of motion tracking. When the quality of the observed data deteriorates due to occlusion, lighting, or other reasons, the prediction weight is automatically increased, relying on historical motion trends to maintain the continuous smoothness of the posture and avoid joint jitter and instantaneous movement.

[0185] In some embodiments provided in this application, the step of determining the target position of each joint at the current time based on the observed position, the confidence level of the observed position and the predicted position of each joint further includes: if the joint position abnormality score is greater than a preset score threshold, determining the target position of each joint at the current time based on the target position of each joint at the previous time.

[0186] Specifically, considering that during the motion-sensing posture estimation process, the observed positions of joints may exhibit severe anomalies due to sudden factors such as limb occlusion, sensor electromagnetic interference, and occasional temporal misalignments of multi-source data, if only observation confidence weight adjustment is used to address such anomalies, when the anomaly amplitude is large enough, the abnormal observation values ​​will still significantly deviate from the final target position, causing joints to instantly jump to positions that do not conform to the laws of human movement. This manifests as skeletal twitching, limb clipping, or discontinuous movements, affecting the smoothness and immersion of motion-sensing interaction.

[0187] Based on this, in some embodiments provided in this application, when the joint position anomaly score is greater than a preset score threshold, the electronic device no longer accepts the observation data at the current moment, but instead uses the target position of the joint point determined at the previous moment as a benchmark to deduce the target position at the current moment. Specifically, the electronic device compares the calculated joint position anomaly score with the preset score threshold. If it determines that the joint position anomaly score is greater than the preset score threshold, it considers that the current observation data has serious distortion, and continuing to include it in the fusion calculation will cause the joint position to jump. Therefore, an anomaly fallback branch is triggered, and the current observation data is directly abandoned from being included in the fusion calculation. Instead, the target position at the current moment is determined based on the target position at the previous moment. In specific implementations, a position preservation strategy can be adopted, that is, the target position at the previous moment is directly used as the target position at the current moment, so that the joint point remains stationary within the anomaly frame; or a uniform speed extrapolation strategy can be adopted, which combines the historical position at an earlier moment to estimate the joint movement speed, and makes a small continuation along the movement direction based on the target position at the previous moment, so that the output transition is more natural and smooth.

[0188] In some embodiments, the target position at the previous moment can be understood as the spatial coordinates of the joint point finally output by the electronic device after the complete anomaly detection and position calculation process in the previous processing cycle.

[0189] In some examples, the target position at the previous moment can be the 3D coordinates of the joint points output by the electronic device in the previous frame. It is understandable that, due to the inertial continuity of human motion, the joint position will not change abruptly between two adjacent frames.

[0190] Thus, in this embodiment, if the joint position anomaly score exceeds a preset score threshold, the target position of each joint at the current moment can be determined based on its target position at the previous moment. This avoids visual defects such as instantaneous joint jumps, skeletal twitching, and limb clipping caused by factors such as occlusion, interference, and instantaneous noise, ensuring the continuity and naturalness of the skeletal animation. Furthermore, determining the target position at the current moment based on the target position at the previous moment fully utilizes the inertial characteristics of human movement, thereby maintaining a smooth transition of motion and avoiding obvious pauses or misalignments caused by abnormal processing.

[0191] To more clearly illustrate the process of acquiring the first skeleton pose data in the embodiments of this application, please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. Also, please... Figure 3 Based on the following exemplary description, that is: In actual operation, factors such as limb occlusion, target out-of-frame movement, and magnetic interference can all cause instantaneous and significant deviations in the observed values ​​of some joints. To avoid this, the embodiments of this application can calculate the confidence level (i.e., the confidence level of the observation position) on a joint-by-joint and frame-by-frame basis, and perform weighted fusion of the observed and predicted values ​​based on the confidence level. When the global anomaly level exceeds a threshold, it automatically switches to recovery mode. Based on the idea of ​​ensuring the stability and continuity of the output skeleton sequence, the acquisition of the first skeleton pose data is completed.

[0192] Specifically, assuming the first j Each joint at any time t The observed value is z t,j The confidence level of the detector output is w t,j ∈[0,1], the short-term predicted value obtained based on historical frame inertial extrapolation is .in, w t,j A value of 1 indicates complete trustworthiness, while a value of 0 indicates complete untrustworthiness.

[0193] Furthermore, confidence-weighted cleaning is performed using formula (5) to achieve soft fusion of observed and predicted values. This allows the observed values ​​to be the primary focus when the confidence level is high, and the predicted values ​​to be gradually transitioned to when the confidence level is low, thus avoiding the jump caused by binary switching.

[0194] (5) In the formula, t,j express t Time of the first j Confidence-weighted cleaning results for each joint; w t,j express t Time of the first j The observation confidence level of each joint (i.e., the observation position confidence level); z t,j express t Time of the first j The original observations (i.e., observation positions) of each joint; express t The first moment is calculated using inertial estimation from the previous few frames. j Short-term predicted value (predicted location) of each joint.

[0195] Considering that joint-by-joint confidence weighting alone is insufficient to deal with the situation where multiple joints fail simultaneously, the overall posture may have deviated from the reasonable range. Therefore, a global anomaly detection mechanism needs to be introduced, and an anomaly score is defined by formula (6).

[0196] (6) In the formula,S t express t Global anomaly score at any given time; α The weighting coefficients for the location residuals are indicated. J Represents the total number of joints; ||·|2 represents the L2 norm; z t,j express t Time of the first j The original observations of each joint; express t Time of the first j Short-term predicted values ​​for each joint; β The weighting coefficients for the velocity residual term; Δz t,j express t Time of the first j The observation speed of each joint (i.e., the difference between adjacent frames); express t Time of the first j Predicted velocity of each joint.

[0197] Understandably, the first term in formula (6) is used to measure the position residual, and the second term is used to measure the velocity mutation residual. The combination of the two terms can improve the sensitivity of anomaly detection. Therefore, even if the confidence label value of a certain joint is high, as long as the deviation between its observed value and the predicted value is large enough, the anomaly score can still effectively trigger the recovery branch.

[0198] Furthermore, when rating S t Exceeding the threshold i rec The recovery mechanism is activated when the time comes, as shown in formula (7).

[0199] (7) In the formula, n t express t The recovery mode trigger flag at any time, 1 for recovery mode, 0 for normal mode; i rec Indicates the threshold for triggering abnormal recovery; express t The final output posture at any given moment; t express t Always trust the normal state after weighted washing; express t The recovery pose generated by the topology constraint predictor at each moment.

[0200] It is also understandable that the posture was restored. It can be generated by the topology constraint predictor, extrapolating reasonable postures based on skeletal structure constraints, and using linear state transition approximation, as shown in Equation (8).

[0201] (8) In the formula, express t The predicted value of the recovered attitude at time step; A Represents a linear state transition matrix; Indicates the output attitude at the previous moment; B Represents the control input matrix; u t express t The control input vector at each time step; or t express t The process noise term at any given time.

[0202] Based on the combined effect of formulas (6)-(8), the original skeleton sequence containing noise and occasional anomalies can be converted into a controlled and continuous clean skeleton sequence, providing a stable input for the subsequent skeleton retargeting module.

[0203] It's also understandable that time alignment only ensures the comparability of multi-source data in the time dimension, but cannot guarantee the validity of the observation data in a single frame. Scenarios such as limb self-occlusion, targets moving out of the camera's field of view, and IMU data jumps caused by magnetic interference can all lead to the corresponding joint's observation data becoming invalid within a short period. Directly inputting such abnormal data into the redirection module will cause severe shaking of the character's posture or non-physiological distortions.

[0204] Based on this, as shown in formulas (6)-(8) above, the embodiments of this application can adopt a two-level processing mechanism. The first level is joint-by-joint confidence soft fusion, that is, joints with higher confidence directly follow the observation value for updating, while joints with lower confidence are replaced by short-term predicted values, thereby achieving stability assurance at the single-joint level. The second level is global anomaly gating, that is, anomaly scores are calculated through position residuals and velocity residuals. S t To reduce the rate of missed anomaly detections. At the same time, a threshold is set. i rec This supports tiered configuration based on gameplay type; for example, competitive scenarios can be configured with more sensitive trigger conditions. (Regarding process noise items...) or t When the recovery branch is triggered, the system does not freeze the character's pose or reset it to the standard pose. T Instead of relying on the previous time step's output state, the attitude is recovered by generating the recovered attitude. It maintains the continuity of action direction and rhythm within the abnormal window through smooth extrapolation. After the abnormality subsides, it can seamlessly switch back to observation-driven mode, and the impact of the entire switching process on the player's perception is kept to a low level.

[0205] In some embodiments provided in this application, step 130 includes: determining the topological mapping relationship between a first joint point in the object skeleton of the target object and a second joint point in the virtual character skeleton; determining target deviation data between the object skeleton and the virtual character skeleton based on the topological mapping relationship; solving the posture transformation relationship between the object skeleton and the virtual character skeleton based on the topological mapping relationship, the target deviation data, and the parent-child bone segment kinematic constraint data of the virtual character skeleton to obtain posture transformation data; and transforming the first skeleton posture data based on the posture transformation data to obtain the second skeleton posture data of the target virtual character. The target deviation data indicates at least one of the following: bone segment ratio deviation, joint rotation angle deviation, bone segment length deviation, and skeleton movement style deviation between the object skeleton and the virtual character skeleton. The parent-child bone segment kinematic constraint data constrains the deviations between multiple bone segments with dependent relationships in the virtual character skeleton.

[0206] Specifically, in motion-sensing game scenarios, the skeletal structure of the player's real body and the skeletal structure of the virtual character in the game belong to two completely different skeletal systems. The player's skeleton is identified by motion-sensing devices and is usually a set of general human key points with fewer joints and a simplified structure. Moreover, there are significant individual differences in body shape, bone segment length, and limb proportions among different players. The virtual character's skeleton, on the other hand, is bound to art assets, equipment attachment points, and special effects anchor points. It has more joints and a more complex hierarchy, and different characters have different body proportions, skeletal axes, and movement styles. If a fixed one-to-one mapping method is used to directly apply the player's posture to the character's skeleton, problems such as limb clipping and hand and foot misalignment due to mismatched bone segment lengths will occur. Anatomical abnormal postures such as elbow retraction and excessive knee bending due to inconsistent joint rotation axes will also occur. End-point position offset caused by the gradual accumulation of long-chain skeletal errors will also result in various problems such as the disconnect between the animation quality and the character's design. These issues not only affect visual performance but also interfere with the accuracy of hit detection in multiplayer battle scenarios, undermining the fairness of the competition.

[0207] Based on this, in some embodiments provided in this application, a topological mapping relationship between the object skeleton and the virtual character skeleton can be established first as the basis for skeleton alignment, determining the matching relationship of corresponding joints in the two skeletons. Then, target deviation data between the two in dimensions such as bone segment ratio, rotation angle, bone segment length, and movement style are calculated. Subsequently, combining the topological mapping relationship, target deviation data, and the kinematic constraint data of the parent and child bone segments of the virtual character skeleton itself, complete posture transformation data is obtained by solving an optimization problem. Finally, the posture transformation data is used to perform spatial transformation processing on the player's first skeleton posture data to generate second skeleton posture data adapted to the virtual character skeleton.

[0208] In some embodiments, the object skeleton can be understood as the player's skeletal structure identified by the motion sensing device, and the skeleton topology diagram formed by joints as nodes and bone segments as connecting edges, which is used to represent the player's current body posture.

[0209] In some embodiments, the first joint point can be understood as a specific joint node in the object skeleton, including but not limited to key human joints such as shoulder, elbow, wrist, hip, knee, and ankle, with each joint point corresponding to position coordinates and rotation information in three-dimensional space.

[0210] In some embodiments, the virtual character skeleton can be understood as the skeletal control system bound to the target virtual character in the game, including the main skeleton that controls the shape of the character and the additional skeletons used to attach equipment and special effects.

[0211] In some embodiments, the second joint point can be understood as a joint node in the virtual character skeleton that corresponds to the player's joint. It is a specific joint in the character's skeletal structure and has a semantic correspondence with the first joint point.

[0212] In some embodiments, the topological mapping relationship can be understood as the corresponding matching relationship between the joint nodes of two sets of skeletons, that is, to clearly identify which joint in the character skeleton a certain joint of the player corresponds to. This relationship ensures that the action semantics are not misaligned before and after mapping.

[0213] In some examples, the player's left elbow joint corresponds to the character's left elbow joint, and the player's right wrist joint corresponds to the character's right wrist joint, forming a one-to-one joint mapping table.

[0214] In some embodiments, the target deviation data can be understood as a set of parameters used to quantify the differences between the object skeleton and the virtual character skeleton in multiple dimensions, including at least one of bone segment ratio deviation, joint rotation angle deviation, bone segment length deviation, and skeleton movement style deviation.

[0215] In some embodiments, the bone segment ratio deviation can be understood as the difference in the relative length ratio of corresponding bone segments within their respective skeletons. It is typically normalized using the root bone segment as a benchmark to eliminate the influence of absolute body size differences among players, ensuring the mapping results are suitable for players of different heights and arm spans. In some examples, using the spinal segment from the hip to the shoulder as a benchmark, the length ratios of each bone segment (upper arm, forearm, thigh, lower leg, etc.) relative to the benchmark bone segment are calculated, and then the corresponding ratios of the two skeletons are compared to obtain the ratio deviation.

[0216] In some embodiments, the joint rotation angle deviation can be understood as the difference in the local rotation reference of the corresponding joints of the two skeletons in a natural standing state. It is used to correct the axial offset when the player's posture rotates to the character's skeletal coordinate system and avoid posture distortion caused by different orientations of the skeleton binding.

[0217] In some embodiments, bone segment length deviation can be understood as the absolute length difference of the corresponding bone segment, used to adjust the spatial position of the limb end and prevent the hands and feet from penetrating the character model or floating on the model surface.

[0218] In some embodiments, skeletal motion style deviation can be understood as a difference parameter in the texture of motion performance, used to make the same player input present differentiated motion styles on different types of characters. In some examples, the style deviation parameter for a heavy armor warrior character will cause the motion amplitude to converge and the rhythm to slow down, showing a heavy and powerful texture; while the style deviation parameter for an agile assassin character will cause the motion amplitude to amplify and the rhythm to speed up, showing a light and sharp texture.

[0219] In some embodiments, the parent-child bone segment kinematic constraint data can be understood as the kinematic constraints between adjacent bone segments with hierarchical dependencies in the virtual character skeleton. This constraint is used to constrain deviations between multiple dependent bone segments, suppress error accumulation in long bone chains, and ensure that while the local rotation of each joint is correct, the end position of the entire limb also conforms to kinematic laws. In some examples, parent-child bone segment kinematic constraints exist between the upper arm and forearm, the thigh and lower leg, and each segment of the spine. The movement of the parent bone segment will cause corresponding positional changes in the child bone segment.

[0220] In some embodiments, the pose transformation relationship can be understood as the mathematical transformation relationship required to transform the pose of an object skeleton into the pose of a virtual character skeleton, which can describe the complete mapping rules from the player's skeletal space to the character's skeletal space.

[0221] In some embodiments, the solution process can be understood as a process of minimizing the topology matching cost, comprehensively considering multiple optimization terms, and calculating the optimal solution through a multi-objective optimization algorithm, ultimately obtaining attitude transformation parameters that can simultaneously satisfy multi-dimensional constraints.

[0222] In some embodiments, attitude transformation data can be understood as a complete set of transformation parameters obtained by solving, including joint mapping matrix, rotation offset parameters, length scaling factor, and style adjustment coefficient, etc.

[0223] In some embodiments, the transformation process can be understood as an operation of performing spatial transformation on the first skeleton posture data joint by joint according to the posture transformation data, projecting the posture of each joint of the player into the virtual character skeleton space, and generating posture data that conforms to the character's skeletal structure.

[0224] In some embodiments, the second skeleton pose data can be understood as the virtual character skeleton pose data obtained after mapping, which is fully adapted to the structural specifications and style settings of the virtual character skeleton and can be directly used as input for subsequent motion recognition and character animation driving.

[0225] To more clearly illustrate the process of acquiring the second skeleton pose data in the embodiments of this application, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. Also, please... Figure 4 Based on the following exemplary description, that is: Understandably, player skeletons and character skeletons differ structurally in terms of the number of joints, bone segment ratios, and joint rotation axes, making direct application impossible. Therefore, it is necessary to establish a topological mapping relationship and pose transformation parameters between the player skeleton and the character skeleton. This ensures that the redirected character pose conforms to skeletal constraints and physical plausibility while preserving the semantics of the actions.

[0226] Based on this, let's assume the player's skeleton diagram is... G p =( V p , E p The character skeleton template is... G a =( V a , E a ), G p and G a The midpoint represents a joint, and the edges represent bone segments. To solve for the joint mapping relationship π: V p V a With attitude transformation parameters M t First, we define the topology matching cost to measure the rationality of the bone segment length and connection relationship after mapping, as shown in formula (9).

[0227] (9) In the formula, π represents the topology matching cost function; π represents the topology mapping relationship from player joints to character joints. Ep. The set of edges representing the player's skeleton; Represents the joints in the player's skeleton u arrive v The length of the bone segment; This represents the scaling factor for the corresponding character's bone segment; This indicates the length of the bone segment corresponding to the mapped joint in the character's skeleton; l e Indicates the illegal topology penalty coefficient; E a Let represent the set of legal edges in the character skeleton. It should be noted that the first term in formula (9) is used to penalize mismatched bone segment lengths, and the second term is used to penalize illegal topological structures generated after mapping, that is, the mapped edges do not exist in the character skeleton.

[0228] Furthermore, considering the significant differences in body shape among different players, directly comparing absolute lengths is not universally applicable. The bone segment lengths are normalized using the root bone segment as a benchmark to obtain dimensionless proportional parameters, thus eliminating the influence of individual body shape differences, as shown in formula (10): (10) In the formula, r uv This represents the normalized length ratio of player skeleton bone segments based on the root bone segment; Indicating player skeleton u arrive v The length of the bone segment; Indicates the length of the player's skeletal root segment; This represents the normalized length ratio of the bone segments in a character's skeleton. Indicating the character's skeleton i arrive j The length of the bone segment; Indicates the length of the root bone segment of the character's skeleton; This represents the bone segment length proportional loss function.

[0229] Meanwhile, at the rotational level, let the local rotation of the player's joints be represented by quaternions. The character's initial bound posture rotation is The offset correction term used to compensate for the difference in rotation between the player and the character in their natural standing posture is: The redirected character joints rotate As shown in formula (11).

[0230] (11) Combining the above formulas (9)-(11), we can obtain the online optimization objective as shown in formula (12).

[0231] (12) In the formula, express t Target attitude transformation parameters at any given time; M t Represents the set of attitude transformation parameters; Indicates the cost of topology matching; Weighting coefficients representing length-proportional loss; Indicates the proportion of bone segment length loss; The weighting coefficients representing rotational losses; Indicates rotational consistency loss; Weighting coefficients representing kinematic loss; This indicates a loss of kinematic consistency between the father and son bone segments; The weighting coefficients representing style loss; This represents the style parameter constraint loss.

[0232] It is worth noting that rotational loss Defined as the geodesic distance between the redirected rotation and the player's desired rotation in quaternion space, it is used to measure the degree of difference in rotation. It can be characterized by formula (13).

[0233] (13) In the formula, This represents the rotational loss function or geodesic distance; J Indicates the total number of joints; arccos (·) represents the inverse cosine function; ·,· This represents the quaternion inner product operation; express t Redirecting rotation of character joints at any moment; This indicates the initial rotation binding of the character's joints; express t The player's joints rotate locally at all times.

[0234] It is also worth noting that formula (12) introduces the loss of kinematic consistency between father and son bone segments. This ensures that the overall position remains accurate while maintaining correct local rotation, thus suppressing the cumulative effect of errors in long-chain skeletons. It can be characterized by formula (14).

[0235] (14) In the formula, Represents the kinematic consistency loss function; E a Represents the set of parent and child joint edges of the character's skeleton; T t,p express t Time Father Joint p The global transformation matrix; T t,j express t Time joint j The global transformation matrix; p→j This represents the expected relative transformation from the parent joint to the child joint; ||·|| F express Frobenius Norm.

[0236] Understandable, This feature can effectively reduce the cumulative torsional deviation of long-chain bones (such as from the spine to the fingertips), avoiding the problem of correct rotation of a single joint but misalignment of the end effector.

[0237] In addition, in formula (12) This is a style parameter constraint used to ensure that the redirection action simultaneously meets the requirements of the player's action intention and the character's style setting, thereby achieving differentiation in the action expressiveness of different character types.

[0238] Additionally, it is understandable that the above exemplary description is used to address motion transfer between heterogeneous skeletons. Specifically, the player skeleton is a simplified, general structure, and different users have significantly different body types. The character skeleton, however, includes art asset-specific structures such as control bones and additional bones, with bone segment proportions and joint definitions different from the player skeleton. If a fixed mapping scheme is used, the deviation is not obvious in static poses, but problems such as joint distortion and end-effector misalignment will occur during dynamic, large-amplitude movements.

[0239] Therefore, through Ensure the structural validity of the topology mapping and clarify the correspondence between player joints and character joints. Length deviations caused by differences in body shape are eliminated through proportional normalization. Rotational space correction is achieved through the bias term. b j Complete. The player's local rotation is now stably projected onto the character's bound space. This bias is not a fixed value; the electronic device can continuously fine-tune it based on accumulated deviations. This achieves parametric control of action style, meaning the same input action exhibits differentiated behavior across different character types. For example, heavily armored characters move with a heavy and steady style, while agile characters move with a light and swift style. The final output character pose balances topological validity and style consistency, and can be used for subsequent processing such as intent recognition.

[0240] Thus, in this embodiment, the posture transformation relationship between the object skeleton and the virtual character skeleton can be solved based on the topological mapping relationship, target deviation data, and the kinematic constraint data of the parent-child bone segments of the virtual character skeleton to obtain posture transformation data. Based on this posture transformation data, the first skeleton posture data is transformed to obtain the second skeleton posture data of the target virtual character, thereby achieving an accurate and stable conversion from the player's motion-sensing posture to the virtual character's skeletal posture. Furthermore, the topological mapping relationship ensures the correct transmission of action semantics, the multi-dimensional target deviation data corrects for differences in body shape, skeletal binding, and style, and the parent-child bone segment kinematic constraints ensure the kinematic rationality of long-chain limbs. Therefore, the final mapped second skeleton posture data effectively restores the player's action intentions and conforms to the skeletal structure specifications and style settings of the virtual character, avoiding joint refraction, limb clipping, and end-effector misalignment, thus improving the visual naturalness and immersion of motion control.

[0241] In some embodiments provided in this application, step 140 includes: determining the action intention of the target virtual character based on the second skeleton posture data and the game context state data of the target virtual character in the target game; predicting the virtual action to be performed by the target virtual character based on the action intention, the second skeleton posture data and the game context state data, and obtaining multiple candidate virtual actions.

[0242] Specifically, relying solely on instantaneous skeletal posture for virtual action recognition is prone to misidentification due to motion noise and posture fluctuations, leading to frequent jumps between different action options. Furthermore, the lack of contextual reference can confuse the meaning of the same posture in different game scenarios. For example, the same outstretched arm posture might correspond to attack and pick-up in combat and interaction scenarios, respectively. Simultaneously, if the electronic device only outputs a single best action guess, subsequent latency compensation, rule verification, or physical correction will lack flexibility. Once the action is deemed unexecutable, no alternatives are available, resulting in response interruptions or abrupt action switching, severely impacting the player's motion-tracking experience.

[0243] Based on this, in some embodiments provided in this application, the electronic device adopts a hierarchical recognition processing method. First, it infers the player's high-level action intent using the redirected second skeleton posture data and game context state data, clarifying the general direction category of the player's current action. Then, based on this intent, it predicts and ranks specific executable virtual actions by combining posture data and context state information, ultimately outputting multiple reasonable candidate virtual actions to form a candidate set for further screening in subsequent stages.

[0244] In some embodiments, the second skeleton pose data can be understood as the pose data obtained by mapping the pose of the real player's human skeleton onto the game's virtual character's own skeletal system after processing such as topological mapping, bone segment ratio correction, rotation offset adjustment, and style parameterization.

[0245] In some examples, when there are significant differences between the player's height and arm span and the game character's, the second skeleton posture data will preserve the player's joint angles, relative limb positions, and other action semantics, while adapting to the character's own bone segment length and skeletal level. This ensures that the meaning of the action is accurately conveyed without joint distortion or limb clipping.

[0246] In some embodiments, game context state data can be understood as a set of various state information reflecting the current situation of the target virtual character in the game, including but not limited to the character's health status, skill cooldown progress, resource balance, scene type, combat stage, relative position with other game objects, and currently held weapons or equipment.

[0247] In some examples, when game context state data shows that an attack skill is on cooldown, the electronic device will not list the skill as a high-priority candidate even if the player's posture is highly similar to the skill's trigger posture, thus avoiding recommending actions that the player cannot currently perform.

[0248] In some embodiments, action intent can be understood as a category of player action purpose summarized at a high-level semantic level, representing the general direction of the action that the player currently wants to perform, in order to narrow the search range of action recognition and improve recognition efficiency and stability.

[0249] In some examples, action intents can include broad categories such as attack intents, dodge intents, interaction intents, and movement intents, with each intent category corresponding to several specific virtual actions.

[0250] In some embodiments, a candidate virtual action can be understood as a set of specific virtual actions that are reasonable under the current posture and context conditions and can be used as alternative execution objects. Each candidate action corresponds to a complete skill segment or animation segment in the game. After the candidate set is sorted from high to low according to the matching degree, it is output to the subsequent delay compensation, rule verification and physical constraint stages for further screening.

[0251] In some examples, candidate virtual actions are usually selected from the top-ranked items in the overall score, such as the top three to top five actions, which ensures a sufficient range of options while avoiding excessive computational overhead due to too many candidates.

[0252] To more clearly illustrate the generation process of candidate virtual actions in the embodiments of this application, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. Also, please... Figure 5 Based on the following exemplary description, that is: After redirecting the player skeleton to the character skeleton and outputting the posture sequence in the character coordinate system, it is necessary to further parse the corresponding action semantics from the continuous posture sequence. However, the same spatial posture may correspond to different action intentions in different contexts. For example, an extended right arm may represent punching in combat mode and picking up an item in interactive mode. Therefore, if single action classification is directly output, it is easily affected by the ambiguity of sensory input, resulting in frequent jumps. Based on this, the embodiments of this application can adopt a hierarchical recognition strategy, that is, firstly, the high-level intention category is determined, then a set of action candidate objects is generated and sorted under the corresponding intention category, and finally, the sorted candidate object set is handed over to the subsequent modules to perform physical constraint and game rule verification.

[0253] Assuming the character pose sequence (i.e., the second skeleton pose data) is Y 1: t And assuming that the context state contains game rule information such as character status, skill cooldown, and remaining resources, it is... x t In terms of the intensity of intent I t = The trigger determination mechanism is described here, which is extended to a probabilistic framework. The intent intensity is encoded into the posterior probability distribution, as shown in formula (15).

[0254] (15) In the formula, Represents actions given an observation sequence and context. a t The posterior probability; a t express t The specific actions at that moment; Y 1:t Indicates from time 1 to t The redirected attitude observation sequence; x t express t The context state at any given moment, such as character status, skill cooldown, remaining resources, and other game rule information; h t express t High-level intention variables at any given moment; H A set of candidates representing the intentions of higher-level entities, such as attack, dodge, interaction, and movement.

[0255] It is worth noting that the high-level intent posterior term adopts an impulse-driven form, and the intent intensity...I t As observational evidence, it directly affects the posterior distribution, thus leading to formula (16).

[0256] (16) In the formula, Let h represent the posterior probability of the higher-level intention. or h Intent h The corresponding intention intensity gain coefficient; I t express t The intensity of intent at any given moment; ψ h (·) represents the contextual feature function of the intention h; H This represents the candidate set of high-level intentions.

[0257] It is worth noting the high-level intention variables h t It can cover coarse-grained action categories such as attack, dodge, interaction, and movement. At the candidate action layer, for each action... a t The comprehensive score is calculated by incorporating the matching probability and various cost items into a unified metric, thus obtaining formula (17).

[0258] (17) In the formula, Score ( a ) indicates an action a Overall score; l p Represents the weighting coefficient of the probabilistic return term; Indicates an action given the intention of a higher level. a The conditional probability; l d The weighting coefficients representing the cost of style bias; D style ( a ) indicates an action a The cost of style deviation is used to measure whether it conforms to the character's design. l r Represents the weighting coefficient of the rule risk item; R rule ( a ) indicates an action a The rule risk item is used to measure whether there is a possibility of violating the rules of combat; l l Weighting coefficients representing latency costs; (a) indicates an action a The expected execution delay.

[0259] Furthermore, the context conditional probability is modeled through state transition and temporal dependencies are introduced to avoid intent jumps caused by independent judgments frame by frame, as shown in formula (18).

[0260] (18) In the formula, express t The posterior probability of the higher-level intention at time step; ∝ is proportional to the sign; This represents the observation likelihood, i.e., the observation given the intention and context. Y t The probability of; The probability of a state transition representing the intention of a higher level; This represents the posterior probability of the intention of the higher-level authority at the previous moment.

[0261] Finally, a temperature-based softmax algorithm is used to generate the probability distribution of candidate actions, and a TopK filter is used to output the candidate set. It is understandable that the temperature parameter... T Controlling the sharpness of the distribution, T The smaller the value, the more concentrated the weight of high-scoring actions. See formula (19) for details.

[0262] (19) In the formula, π t(a) express t Moment Action a Temperature probability distribution; T Indicates temperature parameter; A Represents the set of all candidate actions; K t express t Moment Top - K Candidate action set; TopK(·, K () indicates taking the option with the highest probability. K A function for each action; K This indicates the number of candidate actions.

[0263] In summary, given the inherent ambiguity between motion-sensing actions and game actions, the semantics of the same pose can differ significantly across contexts. If the system only outputs a single optimal action, even minor fluctuations in the input can lead to frequent jumps in classification results. The hierarchical recognition strategy decomposes intent parsing into two levels: high-level intent determination and specific action selection. Each level independently performs probability inference to improve the stability of the recognition results. Therefore, the comprehensive scoring function can incorporate matching benefits and execution costs into a unified quantitative system. Specifically, R rule ( aThis is used to constrain rule risk, automatically lowering the ranking of actions with a high matching probability but a risk of violation. (a) Used to constrain latency costs, so that actions with excessively long execution times are ranked later; This is used to constrain style deviations, lowering the score of actions that do not conform to the character's setting. Through this scoring mechanism, the system outputs not simply the action with the highest matching degree, but the optimal action that takes into account both recognizability and feasibility.

[0264] Additionally, through context state By incorporating information such as the character's current state, skill cooldowns, and resource constraints into the state transition process, the system generates different candidate sets for motion input depending on whether the skill is on cooldown or ready. The system will not recommend actions that are impossible to execute under the current conditions. Ultimately, K sets of candidate actions are output instead of a single result, allowing for adjustments in subsequent physics verification and network synchronization stages.

[0265] Thus, in this embodiment, the action intention of the target virtual character can be determined based on the second skeleton posture data and the game context state data of the target virtual character in the target game. Furthermore, based on the action intention, the second skeleton posture data, and the game context state data, the virtual action to be performed by the target virtual character is predicted, resulting in multiple candidate virtual actions. This can, to a certain extent, reduce misidentification and action jumps caused by instantaneous noise, improving the stability and accuracy of action recognition. Moreover, by introducing the game context state, action judgment can conform to the actual game rules, avoiding the recommendation of unexecutable actions. In addition, by outputting multiple candidate virtual actions, sufficient selection flexibility is provided for subsequent latency compensation, physical constraint verification, and game rule correction, ensuring the continuity and smoothness of action response, ultimately enabling players to obtain a responsive, stable, and gameplay-logical haptic interaction experience.

[0266] In some embodiments provided in this application, the virtual character control method in the game further includes: obtaining time difference prediction and phase difference prediction between the target object performing an action and the target virtual character responding to the target object performing an action; and determining the time difference prediction and phase difference prediction as action control delay information.

[0267] Specifically, considering that the motion-sensing interaction chain involves multiple processing stages from player motion capture, data cleaning, posture calculation, intent recognition to final animation rendering, each stage incurs a certain time overhead, accumulating to form end-to-end motion response latency. If this latency is not effectively estimated and compensated for, players will clearly feel a lag in motion response, and the responsiveness of the motion will be significantly reduced. Furthermore, compensating only for time-related latency is insufficient to cover all experience issues. For periodic or phased actions such as running, combos, and slashes, even if the overall latency is controllable, if there is a phase mismatch between the rhythm of the character's animation and the rhythm of the player's body movements, players will still experience a misaligned sense of impact and / or a feeling of unsynchronized movement.

[0268] Based on this, in some embodiments provided in this application, the electronic device constructs motion control delay information from two dimensions: time difference and phase difference. Specifically, the overall time difference between the target object performing an action and the target virtual character completing the response can be dynamically predicted to obtain an estimated end-to-end delay. Simultaneously, the phase difference between the player's body movement phase and the character's animation phase can be predicted to obtain the degree of misalignment in the action rhythm. Finally, the time difference prediction and phase difference prediction are combined as motion control delay information for the subsequent determination of the target virtual action, enabling the electronic device to simultaneously compensate for both response lag and rhythm misalignment delay problems.

[0269] In some embodiments, time difference prediction can be understood as a dynamic estimate of the overall time lag between the moment the target object makes a haptic action and the moment the target virtual character completes the corresponding action response on the screen. It may include the cumulative latency of the entire link, such as data acquisition, posture cleaning, intent inference, network transmission, and animation rendering, and is used to characterize the end-to-end response lag of haptic control.

[0270] In some examples, the time difference prediction is typically measured in milliseconds and is dynamically updated as network conditions and device load change. For instance, when a home Wireless Fidelity (WiFi) network experiences brief jitter, the time difference prediction increases accordingly, allowing electronic devices to adjust their action look-ahead in advance to ensure that the character's response remains responsive to the player.

[0271] In some embodiments, phase difference prediction can be understood as an estimate of the difference between the phase of the target object's body movement in one action cycle and the phase of the target virtual character animation in its corresponding action cycle, used to characterize the degree of misalignment between the player's action rhythm and the character animation rhythm, and to reflect whether the action beats are aligned.

[0272] In some examples, for periodic actions like running, phase difference prediction can reflect whether the rhythm of the player's strides is synchronized with the rhythm of the character's feet landing. If the phase difference is too large, even if the overall latency is small, the player will feel that the footsteps are off-target, resulting in a weak sense of impact.

[0273] In some embodiments, motion control delay information can be understood as a set of information indicating the delay between the target object performing an action and the target virtual character responding to the action. It consists of time difference prediction and phase difference prediction, including both the overall response delay estimate and the motion rhythm misalignment estimate. As an important constraint parameter for subsequently determining the target virtual action, it provides data basis for delay compensation and phase alignment.

[0274] To more clearly illustrate the process of obtaining motion control delay information in the embodiments of this application, please refer to... Figure 6 , Figure 6 This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. Also, please... Figure 6 Based on the following exemplary description, that is: After obtaining the action candidate set A t Subsequently, there is a cumulative delay between the completion of action judgment and the rendering of the screen. If no compensation is made, the character's action response will lag behind the player's operation, affecting the feel of the action. In addition, for cyclical actions, compensating only for absolute latency is insufficient to guarantee operation consistency. If there is a deviation between the phase of the character's action and the phase of the player's limb movement, even if the end-to-end latency is zero, a sense of rhythm misalignment will still occur. Based on this, the embodiments of this application can solve both action response lag and rhythm phase misalignment through joint control of latency look-ahead prediction and phase synchronization correction.

[0275] Specifically, the total latency is defined as the sum of the overhead of each link in the processing link, as shown in formula (20).

[0276] (20) In the formula, L t express t Total time delay; L cap,t express t The latency of the data acquisition process at any given moment; L clean,t express t The cleaning process is subject to time delays. L infer,t express t Delay in the time inference process; L net,t express t Network latency at any given moment;L render,t express t The rendering delay at any given moment.

[0277] It is understandable that directly using instantaneous delay values ​​for compensation is prone to fluctuations, and short-term jitter in the network or rendering will cause drastic changes in the compensation amount. Therefore, an exponential smoothing term and a queue deviation term are introduced to generate a stable delay prediction value. Specifically, it can be shown in formula (21).

[0278] (twenty one) In the formula, t+1 express t The estimated prediction delay at time +1; clamp (·, L min , L max ) represents a truncation function that restricts the value to [ L min , L max Within the range; β It represents the exponential smoothing coefficient, which controls the mixing ratio of the old and new estimates; L t express t The instantaneous time delay observation at a given moment; t express t The estimated time delay at any given moment; c The gain coefficient representing the queue deviation term; q t express t The length of the queue to be processed at any given time; q ref This indicates a reference value for the queue length. L min This represents the lower bound of the delay estimate; L max represents the upper bound of the delay estimate.

[0279] Furthermore, the number of look-ahead frames for the action is calculated based on the predicted latency. This can be specifically shown in formula (22).

[0280] (twenty two) In the formula, h t express t The look-ahead step (frames) of a given moment. · This indicates the rounding up operation; t+1 This represents the estimated prediction delay. Δt animIndicates the time interval of a single frame of animation; t+ht Indicating foresight h t Predicted action pose after the frame; f pred (·) represents the action look-ahead prediction function; A t express t The set of action candidates at each moment.

[0281] In addition to latency compensation, action beat alignment is also required. Therefore, assuming the player's limb movement position within the motion cycle can be represented by the player's intention phase... The character animation phase, which represents the position of a character animation within the playback cycle, is... The phase error can be expressed as shown in formula (23).

[0282] (twenty three) In the formula, Df t express t Phase difference at any moment (character phase minus player phase); wrap (·) denotes the phase wrapping function, which maps the difference to the range [-π,π]. express t Phase of the character animation at any given moment; express t The phase of the player's body movements at any given moment; L phase This represents the phase loss function.

[0283] It is worth noting that, adopting 1-cos Phase loss function in form L phase It can guarantee continuous differentiability at the 2π period boundary and avoid the error direction from jumping when crossing the 2π boundary, which is of great significance for the beat alignment of cyclic actions such as running and continuous shooting.

[0284] Furthermore, a correction update is performed based on the phase error. This can be specifically shown in Equation (24).

[0285] (twenty four) In the formula, express t Character phase corrected at +1 moment; f t ^{ role}express t The role phase of time; k _ f This represents the phase correction gain coefficient; Dft express t Phase error at any given moment.

[0286] Thus, a joint compensation mechanism based on time delay look-ahead and phase locking is constructed. It is worth noting that the two types of corrections are calculated in a coupled manner within the same control framework, rather than running independently.

[0287] Thus, in this embodiment of the application, the time difference prediction and phase difference prediction between the target object performing an action and the target virtual character responding to the target object performing an action can be obtained, and the time difference prediction and phase difference prediction can be determined as action control delay information. This can cover the overall response lag and phase misalignment of the action rhythm from end to end, ensuring the real-time performance and accuracy of the delay estimation.

[0288] Furthermore, by using latency information in the action determination process, electronic devices can simultaneously perform latency look-ahead compensation and action phase alignment. This solves both the responsiveness problem of characters reacting too slowly and the impact problem of actions not matching the rhythm. This allows players to have a smooth and synchronized motion-sensing interaction experience in both continuous and periodic action scenarios, improving the operation feel and immersion of motion-sensing games.

[0289] In some embodiments provided in this application, step 150 includes: constructing a cost function based on multiple candidate virtual actions, motion control delay information, and motion constraints of the virtual character corresponding to the target game; and solving the cost function to obtain the target virtual action.

[0290] Specifically, while multiple candidate virtual actions obtained through intent recognition may be reasonable in terms of posture matching, they may not meet certain necessary constraints. For example, if the motion control latency factor is not met, the player's actual perceived response lag may exceed the tolerable range, resulting in sluggish movements and poor responsiveness. Furthermore, if the virtual character's motion constraints are not met, physical anomalies such as clipping or feet dangling may occur, or gameplay rules such as skill cooldowns, hit distances, and stun states may be violated.

[0291] Based on this, in some embodiments provided in this application, the electronic device adopts a unified cost function optimization processing method, that is, all relevant elements such as multiple candidate virtual actions, action control delay information and virtual character motion constraints are included in the same cost function for unified modeling. The requirements of different dimensions such as posture following quality, time delay control effect, physical compliance, and rule legality are converted into corresponding loss terms and weighted and combined. Then, by solving the cost function, the candidate action that minimizes the total cost is selected from the candidate set as the final target virtual action to be executed.

[0292] In some embodiments, the cost function can be understood as a mathematical function used to quantitatively evaluate the overall performance of candidate actions. It converts multiple evaluation indicators of different dimensions into calculable loss terms and combines them into a single total value through weighted summation. The smaller the total value, the better the overall performance of the candidate action.

[0293] In some examples, the cost function may include multiple sub-items such as posture loss, latency loss, rule loss, and smoothing loss. Each sub-item corresponds to an optimization dimension, and the weight coefficients configured in front of each item are used to adjust the importance of that dimension in the overall decision. By adjusting the weights, it is possible to flexibly adapt to the experience emphasis of different game types.

[0294] In some embodiments provided in this application, the cost function includes at least two of the following: a posture loss term for constraining stable skeleton mapping, an intent loss term for constraining reliable intent recognition, a latency loss term for constraining controllable end-to-end latency, a rule loss term for constraining legal action gameplay, a smoothing loss term for constraining smooth action timing, and a fairness loss term for constraining controllable trigger differences among different player groups.

[0295] Among them, the posture loss term is used to constrain the stability of the skeleton mapping. It measures the degree of deviation between the character posture corresponding to the target action and the expected posture obtained by mapping the player's haptic input. The greater the deviation, the higher the posture loss. This loss term ensures that the character's actions can accurately follow the player's real haptic input and avoid action deformation or posture deviation.

[0296] The intent loss term is used to constrain the reliability of intent recognition. It measures the degree of matching between the target action and the currently identified player action intent. The more the action matches the player's intent, the lower the intent loss. This loss term ensures that the final action executed is consistent with the player's intention and reduces the probability of false triggering.

[0297] The latency loss term is used to constrain the controllability of end-to-end latency, measuring the overall delay from decision-making to presentation of the target action. The higher the latency, the greater the latency loss. This loss term ensures that the action response speed is kept within the acceptable range for the player, avoiding the impact of action lag on the feel of the action.

[0298] The rule loss term is used to constrain the legality of action gameplay, measuring whether the target action conforms to the physical rules and gameplay rules of the target game. Actions that violate rules such as cooldown time, hit distance, and collision boundary will result in higher rule loss. This loss term ensures that the final action is legal and valid within the framework of the game rules.

[0299] The smoothing loss term is used to constrain the smoothness of the action sequence and measure the continuity of the transition between the target action and the preceding action. Actions with abrupt jumps or hard cuts will produce a higher smoothing loss. This loss term ensures that the action sequence is connected naturally and smoothly, avoiding visual frame skipping or twitching.

[0300] The fairness loss item is used to constrain the controllability of trigger differences among different player groups. It measures the consistency of action trigger conditions among player groups with different heights, arm spans, and operating habits. Excessive differences in trigger thresholds will result in a high fairness loss. This loss item ensures that the motion sensing system's judgment standards for various player groups are relatively uniform, avoiding damage to competitive fairness due to individual player body size differences.

[0301] In some examples, at least two of the aforementioned loss terms can be selected based on the game type and product positioning. The weighting coefficients of each loss term can be adjusted to emphasize different performance requirements. For instance, competitive games can appropriately increase the weights of the rule-based loss term and the fairness loss term, while casual games can emphasize the posture loss term and the smoothing loss term, thus flexibly adapting to different product needs. Understandably, after the cost function is constructed, it is minimized using numerical optimization methods. The solution with the lowest total cost, searched in the candidate action space, is the optimal target virtual action after comprehensively weighing the constraints of each dimension.

[0302] In some embodiments, the solution process can be understood as the process of calculating the constructed cost function and finding the optimal solution, that is, finding the candidate action that minimizes the cost function from multiple candidate virtual actions through mathematical optimization methods, thereby obtaining the target virtual action.

[0303] In some examples, since the candidate set is a finite discrete set, the solution process can be implemented by traversal evaluation, that is, calculating the total value corresponding to each candidate action one by one, and then selecting the candidate with the smallest replacement value as the optimal solution. This method is computationally efficient and produces stable results, which can meet the latency requirements of real-time motion-sensing interaction.

[0304] In some embodiments, motion control delay information can be understood as information indicating the delay between the target object performing an action and the target virtual character responding to the action, including end-to-end time difference prediction and action phase difference prediction, used to measure the degree of lag and rhythm misalignment between the haptic input and the character response, and is the basis for calculating the time delay loss term in the cost function.

[0305] In some examples, motion control latency information can be predicted in real time by electronic devices based on historical latency data, current queue load, and network status, and dynamically updated as the electronic devices operate, ensuring the accuracy of latency assessment.

[0306] In some embodiments, virtual character motion constraints can be understood as various restrictions imposed on the executable actions of virtual characters in the target game, or as the general term for various constraints used to limit the boundaries of executable actions of virtual characters in the process of determining target virtual actions. They are applied to the action selection and decision-making stage to filter infeasible and illegal action options and ensure that the final executed action meets the preset requirements. They can include different types of constraints such as physical constraints and game rule constraints.

[0307] In some embodiments, physical constraints can be understood as restrictions imposed on the actions of virtual characters from the perspective of physical feasibility. These constraints mainly revolve around spatial geometry, kinematic laws, and collision detection, including joint angle restrictions, collision penetration restrictions, foot contact restrictions, and movement speed restrictions. They are used to ensure that the posture and actions of virtual characters appear visually natural and reasonable, and do not exhibit abnormal behavior that violates the common sense of physics.

[0308] In some examples, physical constraints may include constraints such as foot contact with the ground, limbs not penetrating scene objects, and joint rotation angle range. Game rule constraints may include constraints such as skill cooldown status, resource consumption, hit distance, action sequence, and stun status.

[0309] To more clearly illustrate the process of acquiring the target virtual action in the embodiments of this application, please refer to... Figure 7 , Figure 7 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application. Also, please... Figure 7 Based on the following exemplary description, that is: Specifically, this applies to look-ahead / candidate actions for which delay compensation and phase alignment have been completed. However, there may still be situations where physical infeasibility or rule violations exist, such as feet penetrating the ground, triggering during skill cooldown, or hitting at a distance exceeding the skill's effective range. The traditional processing flow involves playing the animation first and then performing post-processing verification and correction. In contrast, the embodiments of this application adopt a joint optimization approach, that is, before the animation is submitted for rendering, the three constraints of motion following accuracy, physical feasibility, and rule legality are incorporated into the same optimizer for unified solution.

[0310] Based on this, the unified solution objective is first defined as shown in formula (25).

[0311] (25) In the formula, u t express t The control input at any given time, i.e., the variable to be optimized; J t express t A unified solution to the objective function at any given time; This indicates a loss in motion tracking, and the goal is to preserve the player's intention in their actions as much as possible. x t express t The role and posture at all times; t+ht Indicates the posture of the target for forward-looking actions; l p Represents the weighting coefficients of physical constraints; Indicates the physical constraint penalty term; l r Represents the weight coefficient of the rule constraint; Indicates the penalty item of the rule constraint Furthermore, physical constraints are expressed in the form of inequalities and are considered hard constraints. Typical constraints include ground penetration constraints and joint torque upper limit constraints, as shown in formula (26).

[0312] (26) In the formula, g floor This represents the ground penetration constraint inequality, used to implement the constraint that feet cannot sink into the ground; y ground Represents ground elevation coordinates; y foot ( x t () indicates posture x t The height coordinates of the lower foot; g torque This represents the upper limit constraint inequality for joint torque, used to constrain joint torque from exceeding a set threshold. t t express t The joint moment vector at time t; ||·||2 represents the L2 norm; t max This indicates the maximum permissible value of joint torque.

[0313] In addition, the rule penalties are based on skill cooldown and hit distance, as shown in formula (27). The remaining rules can be extended in the same form.

[0314] (27) In the formula, C rule This indicates the penalty item imposed by the rule constraint; oh 1 indicates the weight of the cooldown time violation penalty; T cd Indicates skill cooldown time; t last Indicates the time when the skill was last triggered; [ ] + This represents the positive part function, i.e., max(0,z There is a penalty only for violations; the penalty is zero when compliance is maintained. oh 2 indicates the weight of the penalty for violation based on the distance of the hit; d hit ( x t () indicates posture x t The actual hit distance; r skill Indicates the effective hit range of the skill. To balance convergence speed and real-time performance, the augmented Lagrangian method is used to handle mixed-constraint problems, as shown in Equation (28).

[0315] (28) In the formula, L aug Represents the augmented Lagrange function; J t This represents the original optimization objective function; m i Indicates the first i A constrained Lagrange multiplier; g i (·) indicates the first i One constraint function; r This represents the penalty parameter, i.e., the coefficient of the quadratic penalty term.

[0316] During the solution process, a projection update is performed. The projection operator Π_Ω(·) ensures that the output result is within the legal action space that is acceptable to both the animation system and the judgment system, as shown in formula (29).

[0317] (29) In the formula, Indicates the first k +1 iterations of control input; Π_Ω(·) represents the projection operator, which projects the solution into the feasible region Ω; Indicates the first k Control input for the next iteration; or This represents the gradient descent step size, i.e., the learning rate; u L aug This represents the augmented Lagrangian function with respect to control input. u The gradient.

[0318] Understandably, based on the above formulas (25)-(29), motion-driven actions can be upgraded from a visually reasonable level to an online playable level. Specifically, if only the motion reproduction effect is considered while ignoring physical constraints and gameplay rules, problems such as clipping, exceeding the judgment range, and repeated triggering during skill cooldown will frequently occur after the game goes live. In multiplayer battle scenarios, such problems will also directly affect the fairness of the competition.

[0319] Therefore, by incorporating physical constraints and rule constraints into the same optimizer, the traditional sequential processing chain of animation playback, violation detection, and patch correction can be replaced, thereby completing all verification and adjustment before the action is submitted for rendering.

[0320] It is worth noting that in the joint objective function, Used to constrain the deviation between the output posture and the player's original action intention. and Define the feasible domains at both the physical and rule levels.

[0321] It is also worth noting the weighting coefficients l p , l r The value of the weight needs to be determined through an engineering trade-off between the naturalness of the action and the compliance rate. Too low a weight will lead to an increased violation rate, while too high a weight will make the action performance stiff. Therefore, a boundary-sensitive weighting strategy can be adopted, in which the constraint penalty increases as the state approaches the violation boundary, while maintaining a lower weight in areas far from the boundary, in order to balance the naturalness of the action.

[0322] Furthermore, the augmented Lagrange algorithm can handle both hard constraints (such as ground penetration) and soft constraints (such as style preservation) simultaneously, and converges to a feasible solution within a single frame time budget with a small number of iterations. Additionally, the projection operator Π_Ω(·) further ensures that the solution not only mathematically satisfies the constraints but also conforms to the interface specifications between the engine-side animation system and the decision-making system, avoiding situations where the solution is theoretically feasible but practically unenforceable.

[0323] In addition, three types of monitoring indicators can be output simultaneously during implementation, namely the physical violation rate R. phys Rule violation rate R rule and correction deviation E corr These three types of indicators can be used together to determine the direction of parameter tuning. If the violation rate is low but the correction deviation is high, it indicates that the constraint weight is too large, resulting in overcorrection. If the corresponding violation rate is high, the weight of the corresponding constraint item needs to be increased.

[0324] In some embodiments provided in this application, the virtual character control method in a game further includes: obtaining a first action state of a target virtual character in the game client of the target game at the current moment, and a second action state of the target virtual character in the game server of the target game at the current moment; determining state deviation data between the first action state and the second action state; if the state deviation data is greater than or equal to a preset state deviation threshold, controlling the action state of the target virtual character in the game client to change from the first action state to the second action state at the current moment; if the state deviation data is less than the preset state deviation threshold, controlling the action state of the target virtual character in the game client to gradually change from the first action state to the second action state over multiple moments after the current moment.

[0325] Specifically, considering the inherent contradiction between real-time client response and authoritative server judgment in multiplayer online motion-sensing games, if the client waits entirely for the server's response before rendering actions, it will result in excessively high operation latency and poor responsiveness for players. If the client renders entirely based on local predictions without aligning with the server, inconsistencies between client and server states will occur, affecting the fairness of combat judgments and the synchronous multiplayer experience.

[0326] Based on this, in some embodiments provided in this application, the electronic device can first obtain the first action state predicted locally by the client and the second action state recorded by the server, and calculate the state deviation data between the two. When the deviation exceeds a preset threshold, direct synchronization of the current frame is performed to ensure the correctness of the game logic. When the deviation is within the threshold range, it gradually converges through a multi-frame progressive transition, completing the state correction with almost no noticeable impact on the player.

[0327] In some embodiments, a game client can be understood as a game program instance running on the player's local terminal device, responsible for motion input acquisition, local motion prediction and screen rendering, and providing interactive feedback directly to the player.

[0328] In some embodiments, the game server can be understood as an authoritative game logic instance deployed on a cloud server, responsible for maintaining the global game state, executing rule verification, and handling multiplayer synchronization and combat judgment. It is understood that the state data output by the game server has the highest reliability and serves as the benchmark standard for final alignment by all clients.

[0329] In some embodiments, the first action state can be understood as the action state of the target virtual character at the current moment, which is predicted and rendered in real time by the game client based on local motion input. It includes multi-dimensional state information such as the rotation angle of each joint of the character, the spatial position of the root node, the current action execution stage, and the skill progress. It is the prediction result calculated locally by the client.

[0330] In some embodiments, the second action state can be understood as the standard action state of the target virtual character at the same moment, determined by the game server after authoritative verification. It is an authoritative judgment result obtained after comprehensively considering factors such as physical constraints, gameplay rules, and multi-player synchronization logic, and serves as a reference benchmark for the alignment of all client states.

[0331] In some embodiments, state deviation data can be understood as a quantitative measure of the degree of difference between the first action state and the second action state. In some examples, state deviation data can be calculated using a weighted Euclidean norm, assigning higher weights to key dimensions such as root node position and skill status, and lower weights to secondary dimensions such as distal joint angles. This ensures that the deviation value objectively reflects the degree of inconsistency actually perceived by the player. For example, the weights of the character's root node world coordinate deviation and the current skill identifier deviation are significantly higher than the weight of a single finger joint angle deviation, ensuring that the deviation score is positively correlated with the intensity of the player's subjective perception.

[0332] In some embodiments, the preset state deviation threshold can be understood as a pre-set deviation boundary reference value, used to distinguish between two synchronization scenarios: small deviation and large deviation. This threshold can be configured differently according to the game type and motion sensitivity. Competitive battle games can set relatively strict thresholds to ensure fairness, while casual motion-sensing games can appropriately relax the threshold to improve visual smoothness.

[0333] In some embodiments, multiple moments can be understood as multiple consecutive rendering frame cycles during which the client completes the gradual recovery process. The recovery rate can be controlled by an incremental coefficient to make the transition process exhibit a convergence characteristic of being slow at first and fast later, reducing the abruptness in the player's vision.

[0334] To more clearly illustrate the execution process of client-side action state synchronization in the embodiments of this application, please refer to the following exemplary description: Specifically, in multiplayer online game scenarios, there is a conflict between the client's need for real-time response and the server's requirement for authoritative consistency. Specifically, if rendering is only done after server confirmation, significant operational delays will occur; if client prediction is relied upon entirely, global state consistency cannot be guaranteed. To address this, this application's embodiments employ a tiered correction strategy: gradual smooth correction is performed when the deviation is within acceptable limits, and state rollback is triggered when the deviation exceeds a threshold or a rule conflict occurs, thereby maintaining client-server state consistency while ensuring smooth operation.

[0335] Based on this, we first assume that the client predicts the state as follows: The server's authority status is The end-to-end consistency deviation is defined as shown in formula (30).

[0336] (30) In the formula, e t express t Inconsistency deviation between client and server at any given moment; express t The client's predicted state at any given time; express t The server's authority status at any given moment; ||·|| W represents the weighted norm operation; W represents the weighted diagonal matrix of the state dimensions. It is worth noting that matrix W assigns differentiated weights to different state dimensions, with key dimensions such as root node position and skill state having higher weights than secondary dimensions such as single joint angles.

[0337] When the deviation is within the allowable range, a gradual correction strategy is adopted to distribute the error over subsequent frames and gradually eliminate it, as shown in formula (31).

[0338] (31) In the formula, express t + k Client status after real-time correction; express t + k The original client-predicted state at any given moment; α k Indicates the first k The frame's sync coefficient; express t The server's authority status at any given time; express t The client's predicted state at any given time; k This indicates the number of frames from the starting point of the correction. t s This represents the time constant for smooth recovery, used to control the recovery speed.

[0339] When the deviation exceeds the threshold, the confirmation delay exceeds the limit, or a rule conflict occurs, the state rollback is triggered, as shown in formula (32).

[0340] (32) In the formula, I rollback This indicates the rollback trigger indicator function; a value of 1 triggers a rollback, while 0 does not. e t This indicates a deviation in consistency between client and server. e e Indicates the allowable deviation threshold; Δt ack This indicates the deviation in the round-trip time of the confirmation package; e tThis indicates the allowable threshold for time deviation; I rule_conflict This indicates a rule conflict flag; a value of 1 indicates that a conflict exists.

[0341] In some embodiments, the rollback depth is determined in an adaptive manner, and the optimal balance between correction accuracy, visual continuity and computational overhead is achieved based on the cost function, as shown in formula (33).

[0342] (33) In the formula, J rb ( r ) indicates the rollback depth is r The cost function at that time; r Indicates the rollback depth, i.e., how many frames are rolled back; i Indicates the first i The client-predicted state of the frame; Indicates the first i The server authority status of the frame; l vis Weighting coefficients representing the cost of visual abrupt changes; D vis ( r ) indicates the rollback depth r The corresponding degree of visual abruptness; l cpu Weighting coefficients representing the CPU computation cost; r Indicates the rollback depth; r max Indicates the maximum allowable rollback depth It is understandable that formula (33) is designed in accordance with the principle of minimum intervention, that is, smooth correction is preferred, rollback is only performed when necessary, and the rollback depth is controlled to the minimum value that meets the consistency requirements.

[0343] It is also understood that in this embodiment of the application, the client predicts and renders based on local input to ensure real-time response, the server performs authoritative verification synchronously and returns the correction status, and when the deviation between the client and server is small, it is smoothly corrected in a gradual manner, and when the deviation is large or there is a rule conflict, the state is rolled back and replayed.

[0344] The reason for using a weighted norm for deviation measurement is that different state dimensions have varying degrees of impact on player perception. Root node world coordinate offsets or inconsistencies in skill states are easily noticed by players, while small deviations in single joint angles usually do not affect the overall experience. Increasing the weight proportion of key dimensions through the weight matrix W makes the rollback trigger conditions more closely match the player's actual perception.

[0345] For small deviations, a coefficient can be used. α k The incremental, smooth correction path allows errors to gradually converge over subsequent frames, avoiding visual jumps caused by single-frame correction. For large deviations or rule conflicts, the system performs a state rollback. During the rollback process, the rollback depth... r It is not a fixed value, but is determined by optimizing the solution to take into account three objectives: the overall correction effect, visual continuity, and computational resource consumption.

[0346] Furthermore, a unified circular buffer is typically maintained during implementation to cache input data, animation states, collision states, and critical decision events. Additionally, rollback operations require simultaneous reconstruction of the action sequence and decision results to ensure consistency between visual presentation and logical judgment, avoiding a disconnect between visual state and damage assessment.

[0347] Thus, in this embodiment, the first action state of the game client and the second action state of the game server can be obtained, the state deviation data between the two can be determined, and the current frame can be directly synchronized or multiple frames can be gradually changed according to the relationship between the deviation and a preset threshold. This allows for the organic unification of real-time client response and authoritative server judgment through a differentiated synchronization strategy based on deviation grading. A gradual correction strategy is adopted for the vast majority of normal frames to ensure visual smoothness and responsive operation, avoiding frequent character jumps and twitches. Forced synchronization is only executed when the deviation accumulates to a level that may affect the fairness of the game, ensuring the authority and consistency of the gameplay rules. This solution effectively breaks the traditional conflict between smoothness and accuracy in synchronization schemes. In the high real-time scenarios of motion-sensing games, it maintains millisecond-level operational feedback while controlling the difference between client and server states within a reasonable range, significantly reducing judgment disputes in multiplayer battles. It also has good scene adaptability, allowing for flexible adjustment of the synchronization strategy according to different game types.

[0348] In some embodiments provided in this application, considering that after a game is launched, there are usually updates such as new characters, maps, and weapons, as well as changes in the composition of the player group, these changes may cause the initial debugging parameters to gradually deviate from the optimal state over time. In order to further ensure the robustness of the target game's continuous operation, an online iteration mechanism based on operation logs can be established. Specifically, a closed loop can be formed by revenue-driven parameter optimization and risk-constrained gray-scale management to ensure that the system maintains stable performance and experience quality during long-term operation.

[0349] Specifically, a window-level comprehensive benefit function can be defined first to integrate multi-dimensional evaluation indicators into a single scalar, thereby uniformly measuring the optimization effect. The window-level comprehensive benefit function can be shown in formula (34).

[0350] (34) In the formula, R t express t The window's overall benefit score; w 1~ w 5 represents the weighting coefficient of each indicator item; Q t This represents the experience quality score, which is derived from a scoring model or A / B index; L t Indicates latency; E t Indicates the motion error index; C t Indicates the rate of abnormal complaints; U t This represents the fairness penalty value.

[0351] Furthermore, the parameters in formula (34) can be updated along the gradient direction of the comprehensive return, which can be expressed in the form of formula (35).

[0352] (35) In the formula, i k+1 Indicates the first k +1 updated parameters; i _ k Indicates the first k The parameter value of the next time; or This indicates the parameter update step size, i.e., the learning rate; θ Indicates the parameter i Find the gradient; [ R t | i k ] indicates a given parameter i k Expected returns at that time.

[0353] Understandably, to ensure the robustness of parameter release, parameters must undergo gray-scale verification before release and must meet the upper bound constraint of risk. Based on this, three types of operational risk events can be set, and the release process will be terminated when the probability of any of these three types of operational risk events exceeds the threshold. Specifically, as shown in formula (36).

[0354] (36) In the formula, Pr (·) indicates probability operations; Crash This indicates the change in the crash rate; dc This indicates the acceptable threshold for changes in the crash rate; Dunfair Indicates the change in fairness indicators; d f This indicates the acceptable threshold for changes in fairness. Δrb This indicates the change in the rollback rate; d r This indicates the allowable threshold for changes in the rollback rate; α This represents the upper bound probability of risk, i.e., the risk tolerance for gray-scale release.

[0355] The new strategy uses the Lower Confidence Bound (LCB) to determine the scaling decision. This indicator is adaptive to the sample size and automatically tends to be conservative when the sample is insufficient, as shown in formula (37).

[0356] (37) In the formula, LCB (Δ R ) represents the lower confidence bound of the change in returns; Δ A sample mean estimate representing the change in returns; z 1-β Represents the 1- of the standard normal distribution β Quantiles are the coefficients corresponding to confidence levels. s ΔR The sample standard deviation representing the change in returns; n Indicates the number of samples.

[0357] Understandably, when LCB (Δ R When ) < 0, the parameters can be automatically rolled back to the previous version.

[0358] In addition, to address the degradation of model performance caused by changes in the distribution of input data, a distribution drift detection mechanism is introduced, as shown in formula (38).

[0359] (38) In the formula, D KL (·‖·)express KL Divergence is used to measure the difference between two probability distributions. p t ( x This represents the data distribution at the current moment; p _0( x () represents the distribution of the baseline data during model training; t _ drift This indicates the threshold for triggering distribution drift.

[0360] When the KL divergence exceeds a threshold, it indicates that the current data distribution has significantly deviated from the baseline distribution during model training. Continuing to use the original parameters will lead to a continuous accumulation of errors. At this point, high-risk parameters can be frozen to control the scope of the impact, and then the retraining process can be initiated.

[0361] In summary, by using a unified comprehensive benefit function, the R&D team can balance multiple dimensions of demands such as experience quality, operational performance, and competitive fairness within the same objective framework, avoiding situations where one aspect is neglected in multi-objective optimization.

[0362] Meanwhile, parameter updates can be implemented through offline training to generate candidates, online small-scale verification, and tiered gradual rollout, thereby ensuring continuous optimization while keeping potential risks within an acceptable range. The risk constraints set for three types of events—increased crash rate, fairness deviation, and increased rollback rate—constitute the operational safety baseline. If any indicator exceeds the limit, rollout should be immediately halted to prioritize the stability of the entire server.

[0363] The lower confidence bound mechanism automatically adjusts the conservatism of decisions based on sample size. When the sample size is small, the confidence interval is wider, the lower bound is lower, and the scaling conditions are more stringent. As the sample size accumulates, the confidence interval narrows, allowing for a gradual increase in traffic volume. This mechanism effectively avoids misjudgments caused by short-term random fluctuations and prevents optimization decisions from being shelved for extended periods due to insufficient samples. Automatic rollback is triggered when the lower confidence bound is less than zero, enabling automated execution of policy control and reducing reliance on manual judgment.

[0364] Distribution drift detection is used to identify scenarios where structural changes occur in game content or user groups. When the KL divergence exceeds a threshold, high-risk parameters are first frozen to curb error propagation, and then the retraining process is initiated. After the parameters trained based on the new distribution are validated, the grayscale output is resumed. This mechanism avoids the cumulative error effect of online learning mechanisms during periods of abrupt data distribution changes, such as version updates.

[0365] In some embodiments provided in this application, considering that some complex gameplay has higher requirements for the depth of action understanding and the ability of automatic parameter evolution, hierarchical temporal coding, three-dimensional mesh recovery constraints and multimodal reinforcement learning strategy updates can be introduced to further support advanced application scenarios such as long-range action understanding and online adaptation on the basis of completing the basic motion capture to animation mapping process.

[0366] Firstly, please refer to Figure 8 , Figure 8This is a flowchart illustrating the virtual character control method in a game provided in this application embodiment. Specifically, in the process of recognizing action intent based on the second skeleton pose data, the representational capability of the temporal encoder directly determines the recognition accuracy. Therefore, to further improve the recognition accuracy, this application embodiment can adopt a hierarchical coding architecture, using the collaborative operation of two types of modules—Long Short-Term Memory (LSTM) and Transformer—to achieve feature extraction at different time scales. LSTM is used to capture inertia and velocity changes within a short window, while Transformer is used to model long-range dependencies across windows.

[0367] In the process of recognizing action intent, short window temporal information is first aggregated by LSTM, and then input into Transformer for global association modeling. This process can be expressed by formula (39).

[0368] (39) In the formula, Indicates the first l Layer LSTM in t The hidden state at any given moment; LSTM _ l (·) indicates the first l Layer LSTM unit operations; x t express t Input at any moment; Indicates the first l The hidden state of the LSTM layer in the previous time step; H t express t Long-range temporal characteristics of the Transformer output at any given time; Transformer(·) represents the Transformer encoder operation; w Indicates the length across the window; L express LSTM The total number of floors.

[0369] Furthermore, H t After projecting onto the intention space, the posterior probability is calculated as shown in formula (40).

[0370] (40) In the formula, z t express t The feature vectors projected onto the intention space at each moment; W h Represents the projection weight matrix; H t expresst Long-range temporal characteristics of the Transformer output at each time step; b h This represents the projection bias vector.

[0371] Understandably, based on the hierarchical coding architecture, short-term dynamic stability and long-term semantic consistency can be integrated into the same posterior estimation framework, thereby effectively reducing misjudgments caused by single-frame noise during continuous techniques and action transitions.

[0372] Secondly, in the process of determining the second skeleton pose data using the first skeleton pose data, or in other words, in the process of completing the retargeting calculation based on the skeletal topology, the rationality of the skeletal pose is not necessarily equivalent to the rationality of the visual effect. That is, even when the joint angles are within a reasonable range, visual anomalies such as interlacing or collapse may still appear on the character's skin surface. Therefore, in some embodiments of this application, a mesh-level geometric verification mechanism is introduced, that is, through the skeletal pose... q t With body shape parameters s t Decoding generates the human body mesh vertices, as shown in formula (41): (41) In the formula, V t express t The human body mesh vertices are decoded from the skeleton posture in real time; D ψ(·) The parameter is ψ The grid decoder; q t express t The skeletal posture at any given moment; s t Indicates body shape coding parameters; L mesh Represents the mesh consistency loss; Π(·) represents the projection operator, used to transform 3 D Vertex projection to 2 D Image plane; K t express t Key points of two-dimensional observation at any given moment; l n Indicates the weight of the normal constraint term; N (·) represents the mesh normal calculation function; t express t Reference normal at any given moment.

[0373] Mesh consistency loss L mesh Incorporating this into the aforementioned formula (12), we obtain formula (42).

[0374] (42) Understandable, L mesh The constraints introduced can filter out bone poses in advance during the retargeting stage, but there are abnormal deformations on the mesh surface, which improves the final visual quality and the quality of the second skeleton pose data.

[0375] Thirdly, in the process of updating parameters based on gradient descent as shown in formula (35), considering that the game environment has multi-dimensional coupling characteristics, simple gradient optimization is difficult to balance all dimensions. Therefore, in order to improve this situation, a reinforcement learning framework is introduced to realize structured parameter decision-making.

[0376] First, define the state. This includes timing characteristics, phase error, end-service deviation, and rule margin. Redefining the action. a t Let be the adjustment vector for the threshold and weights. The advantage function is used for policy update, as shown in formula (43).

[0377] (43) In the formula, Â t express t The advantage function estimate at time 1; Q ω (·) indicates that the parameter is oh Action value function Q ; s t express t The state at any given moment, such as timing characteristics, phase error, end service deviation, rule margin, etc.; a t express t The action at any given time, i.e., the threshold and weight adjustment vector; V ω (·) indicates that the parameter is oh State value function V ; i Indicates the policy network parameters; or Indicates the policy update step size; θ logπ θ (·) represents the policy log probability with respect to parameters. i gradient; π θ (·) indicates that the parameter is i The policy network is used to output the action probability distribution.

[0378] Furthermore, by introducing security gating execution conditions, the parameter adjustments obtained from policy learning must simultaneously pass intent security verification and value assessment before they can take effect. This can be specifically illustrated in formula (44).

[0379] (44) In the formula, express t The parameter adjustment action that is finally executed at any given moment; D (·) represents a gating indicator function, which is 1 if the condition is met, and 0 otherwise; Indicates the strength of the intent after the security correction; i t Intent safety threshold; Q ω ( s t , a t () represents the value assessment of a state-action pair; t Q Indicates the safety threshold for value assessment; a t This indicates the adjustment action to the output of the original strategy.

[0380] Understandably, the LSTM module in the first aspect focuses on short-term dynamic feature extraction. By encoding the joint velocities and accelerations of the most recent frames, it suppresses the impact of single-frame jitter and outputs a summary of stable motion features within a short window. The Transformer module then performs cross-window temporal correlation modeling to distinguish between segmented movements in continuous techniques and unintentional random limb movements. This two-level encoding concatenated structure improves the robustness of recognizing continuous action sequences and transition phases.

[0381] Furthermore, the 3D mesh restoration module in the second aspect above serves as a supplementary verification step for skeletal redirection. It does not replace the original skeletal mapping process but rather provides consistency supervision at the surface geometry level. The skeletal level verifies the topological and kinematic rationality, while the mesh level further verifies the visual rationality of the skin surface morphology. Together, they ensure the quality of the redirection output.

[0382] Furthermore, the multimodal reinforcement learning module in the third aspect mentioned above does not directly modify the game's decision-making rules. Its role is to adaptively fine-tune the thresholds and weights within safety constraints. Only when the update simultaneously satisfies the conditions of positive returns and controllable risk will the corresponding parameter adjustments be allowed to enter the online environment.

[0383] To more clearly illustrate the virtual character control method in the game provided in the embodiments of this application, please refer to... Figure 9 , Figure 9This is a flowchart illustrating a virtual character control method in a game provided in an embodiment of this application. Specifically, as shown... Figure 9 As shown, the virtual character control method in the game provided in this application embodiment can be implemented through 12 modules / steps, namely: Multimodal input acquisition: Acquire multi-channel motion-sensing input data from the target object, including heterogeneous signals from multiple sources such as camera skeleton point stream, IMU inertial data stream, and controller event stream. This step provides the raw input for the entire system, and the sampling rate, timestamp accuracy, and frame drop patterns vary among different devices.

[0384] Unified clock alignment: Mapping multiple inputs to a unified time axis eliminates clock domain differences between devices. Specifically, time scale standardization is achieved through affine mapping, then clock drift is recursively estimated and continuously compensated using reference pulses, and finally, irregular time point data is resampled to a unified time axis. t The grid is used to output the fused sequence. This step is used to resolve the misalignment of the time axes of the various data streams, establishing a stable time base for all subsequent calculations.

[0385] Confidence cleaning and anomaly recovery: Quality control is performed on the skeleton data after unifying the time base. Confidence is calculated joint by joint using confidence-weighted soft fusion. Specifically, high-confidence data follows the observation, while low-confidence data reverts to short-time predictions. Simultaneously, a global anomaly score is calculated, and the position and velocity residuals are combined to determine whether a recovery branch is triggered. When triggered, a smooth fallback pose is generated by the topology constraint predictor to avoid violent character jitter or instantaneous movement. This step outputs continuous and stable first skeleton pose data.

[0386] Topology-adaptive redirection: This process maps the player's skeleton to the game character's skeleton, translating human language into the character's language. The first step is to solve for the joint topology mapping relationship. Next, bone segment length ratios are normalized to eliminate body shape differences. Then, rotational space offset correction is performed. Finally, style parameters are injected, and the optimal pose transformation is solved by combining parent-child bone segment kinematic constraints. The final output is the second skeleton pose data, ensuring that the meaning of the action is not lost and the physical pose is reasonable.

[0387] Hierarchical Intent Recognition: Based on the redirected posture sequence, action semantics are recognized using a two-layer architecture of high-level intent and candidate actions. First, high-level intent categories such as attack, dodge, interaction, and movement are inferred. Then, multiple candidate actions are generated under this intent and scored comprehensively. The scoring dimensions include four aspects: probability confidence, style bias cost, rule risk, and expected latency. Finally, a TopK candidate set is output instead of a single answer, thus reserving flexibility for subsequent corrections.

[0388] Latency compensation and phase alignment: This addresses the latency issues of late-arriving actions and out-of-beat rhythm. Specifically, it decomposes the latency of each link in the chain and obtains stable prediction values ​​through smoothing estimation and queue deviation perception, thereby calculating the look-ahead step size and generating the look-ahead posture. Simultaneously, it extracts the player's intention phase and the character's animation phase, calculates the phase difference, and continuously corrects it through the controller. Finally, it outputs a low-latency and beat-aligned target action.

[0389] Joint Correction of Physics Rules: Before animation submission for rendering, the physical feasibility and gameplay legality are uniformly verified. A joint optimization objective is constructed, including trajectory tracking items, physical constraints (such as ground penetration, collision, and torque limit), and rule constraints (such as skill cooldown, hit distance, and resource window). The augmented Lagrangian method is used to solve the objective and project it onto the feasible region to ensure that the output is both in line with the player's intention and a legal trajectory that can be played by the engine, thereby avoiding online faults such as clipping, exceeding the judgment range, and cooldown triggering.

[0390] Server-based authoritative verification: In multiplayer scenarios, the server makes an authoritative determination, performing rule-based and physical secondary verification on the key status summaries uploaded by the client, and returning an authoritative status and confirmation information. This module ensures fairness in the competition; client predictions are only used for immediate feel, and the final judgment is based on the server's result.

[0391] Network synchronization and rollback coordination: Handling differences in client / server states employs a tiered strategy of smooth correction for small deviations and controlled rollback for large deviations. Weighted consistency deviation is calculated, with higher weights given to key dimensions such as root node position and skill status. When the deviation is less than a threshold, the error is distributed across multiple future frames for gradual correction, resulting in near-imperceptible performance for players. When deviations exceed limits or rule conflicts occur, the optimal rollback depth is adaptively selected based on a cost function, striking a balance between correction effectiveness, visual abruptness, and CPU consumption.

[0392] Experience and Fairness Metrics Dashboard: This dashboard summarizes end-to-end operational metrics, including end-to-end latency distribution, false trigger rate, rollback rate, physical violation rate, rule violation rate, and trigger differences among different body types, forming a unified monitoring view. This module provides data input for online learning and serves as a direct basis for the operations team to assess system health.

[0393] Online learning and operational closed loop: Transforming operational logs into fuel for continuous optimization. Constructing a comprehensive revenue function to uniformly balance multiple dimensions of objectives such as experience quality, latency, error, complaint rate, and fairness, generating candidate parameters along the revenue gradient; verifying the effect through A / B testing, setting three risk thresholds: crash rate, fairness deterioration, and a surge in rollback rate, with automatic rollback when these thresholds are exceeded; simultaneously monitoring data distribution drift, freezing high-risk parameters and triggering retraining when anomalies occur.

[0394] Parameter hot update and version management: Parameters that have completed gray-scale verification are injected into the runtime pipeline via hot update, supporting fine-grained gray-scale releases and rapid rollbacks by version, device, and user group. The output of this module flows back to multiple front-end modules such as latency compensation, physical rule correction, and network synchronization, forming a complete closed loop of operation, monitoring, learning, and updating, ensuring that the system continues to evolve and does not degrade in long-term operation.

[0395] In addition, Figure 9 In this context, the skill state machine resource system can be understood as a game context module independent of the main link. It provides rule information such as character state, skill cooldown, and resource availability to the hierarchical intent recognition and physical rule joint correction, ensuring that action recognition and rule constraints are always consistent with the actual game state. Parameter hot update feedback can be understood as the simultaneous feeding of the parameter hot update and version governance module outputs into three nodes: latency compensation and phase alignment, physical rule joint correction, and network synchronization and rollback coordination. This enables dynamic control of the front-end execution link based on the learning results from the operations side.

[0396] To better illustrate the implementation and beneficial effects of the virtual character control method in the game according to the embodiments of this application, please refer to... Figure 10-12 , Figure 10 and Figure 12 These are all schematic flowcharts illustrating the virtual character control method in a game provided in the embodiments of this application. Figure 11 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application.

[0397] Firstly, please refer to Figure 10 The content shown is as follows. Figure 10 This application describes the application of the virtual character control method provided in this embodiment of a game in a two-player real-time fighting game. Specifically, the application scenario targets a two-player real-time fighting game, and the deployment platform covers cross-platform online mode between console and personal computer (PC) terminals. During the testing phase, it was found that when players perform three consecutive actions—rapid punching, blocking, and sliding—the character's performance frequently exhibited inconsistent phenomena, such as timely action response but delayed judgment, or accurate judgment results but jittery animation.

[0398] It's worth noting that this situation isn't caused by insufficient precision in a single algorithm, but rather by the lack of a unified closed loop between the input time base, skeletal mapping, skill rules, and network synchronization. In a competitive gaming environment, even an extra 20 to 40 milliseconds of latency jitter can be perceived as input response delay by players, directly impacting the gaming experience.

[0399] Based on this, applying the virtual character control method in the game provided in this application embodiment to this scenario, the following steps can be performed: Multi-source acquisition and temporal alignment: The acquisition layer synchronously accesses camera skeletal keypoints, IMU angular velocity data, and controller event inputs, first unifying the timelines of the multi-source data. Unaligned multi-timeline data introduces phase noise, leading to a decrease in the recognition accuracy of subsequent classification models.

[0400] Confidence cleaning and anomaly reconstruction: After clock unification is completed, confidence cleaning is performed. Occluded joints are repaired using soft reconstruction instead of directly discarding the corresponding frame data. This mechanism can prevent interruptions in action continuity when players turn quickly or when limbs self-occlude.

[0401] Topology Adaptive Redirection: This module is used to map the player's skeleton to the character's skeleton. Specifically, it can dynamically adjust the bone segment ratio and rotation offset parameters to eliminate the judgment distance deviation caused by the difference between the player's and character's skeletons of different body types, thus ensuring the fairness of the competition.

[0402] Intent impulse recognition: The intent impulse layer does not directly output the final skill judgment result. Instead, it first generates a sequence of intent intensity within a short time window, and then passes it to the skill state machine for executability screening. This layered design can effectively reduce false triggering caused by single-frame noise.

[0403] Joint rule and physics verification: The joint rule and physics verification module incorporates the hit decision box, stun state, physical value and collision feasible region into the same constraint solver. Actions that do not meet the constraints will be projected to the nearest feasible solution.

[0404] Client-server collaborative synchronization: The client adopts a predictive mechanism to ensure real-time response. After the server sends back authoritative results, it performs partial rollback or smooth correction based on the degree of deviation, ultimately achieving the dual goals of local real-time feedback and fair and consistent gameplay.

[0405] Rendering layer input isolation: The rendering end only receives motion trajectory data that has been validated by rules. This layered design facilitates problem localization and can clearly distinguish whether the fault originates from the input layer, the decision layer, or the network layer.

[0406] After applying the virtual character control method in the game provided in this application embodiment to this scenario, and after a 1v1 internal battle stress test, the end-to-end action feedback latency decreased from the average of 96.4 milliseconds in the original scheme to 71.8 milliseconds, and the 95th percentile latency decreased from 134.2 milliseconds to 89.5 milliseconds. Based on the latency decomposition of formula (20), the end-to-end total latency expression shown in formula (45) can be obtained.

[0407] (45) In the formula, L e2e Indicates the total end-to-end delay; L cap Indicates the delay in the data collection process;L clean This indicates the delay in the cleaning process; L map This indicates the time delay in the skeleton mapping process; L intent This indicates a delay in the intent recognition process; L render Indicates the latency of the rendering process. The decomposition results show that the main benefit of latency optimization comes from the mapping stage. L map With intent recognition L intent The fluctuations converge, rather than simply optimizing network transmission.

[0408] Regarding the fairness of the competition, the differences in the probability of action triggering are statistically analyzed by grouping by body type, as shown in formula (46).

[0409] (46) In the formula, D fair This indicates the degree of difference in action triggering between body size groups, i.e., a fairness indicator. g i , g j Indicates different body types grouped together; Pr ( a | g i ) indicates body type grouping g i Next action a The trigger probability.

[0410] It should be noted that after applying the virtual character control method in the game provided in the embodiments of this application, D fair The error rate decreased from 0.118 to 0.041. In other words, the proportion of player misjudgment complaints that were judged as hits even though a block had been executed decreased, indicating that the dual-ring architecture improved the interpretability of the judgment results and player trust while reducing latency.

[0411] Therefore, for the development process, the animation team does not need to frequently add keyframes to address network jitter. Furthermore, gameplay mechanics can be developed for skills and levels based on a unified intent interface. In addition, the online engineering team has obtained a stable benchmark for determining rollback triggers.

[0412] Secondly, please refer to Figure 11 The content shown is as follows. Figure 11This description illustrates the application of the virtual character control method provided in this application within an action role-playing game (ARPG). Specifically, a particular application scenario corresponds to an open-world ARPG project, where a three-player cooperative mechanism is implemented within a dungeon: one player is responsible for parrying, one for pulling, and one for charging and finishing off enemies. After the motion-sensing interaction version was rolled out in a limited release, two typical failures occurred: first, semantic conflicts existed between the actions of the three players simultaneously on the server side, leading to abnormal chain skill triggering order; second, during network jitter, the client side displayed that the cooperative action was completed, but the server side determined that the cooperative conditions had not been met, resulting in a discrepancy in player experience.

[0413] Based on this, the virtual character control method in the game provided in the embodiments of this application can be applied to this scenario to extend the intention pulse mechanism from single-person recognition to multi-person group collaborative arrangement, and incorporate the time window constraints, team role division and BOSS phase status in the dungeon mechanism into the decision-making system.

[0414] Specifically, such as Figure 11 As shown, the motion inputs of the three players first converge to the client-side collaborative orchestrator. The orchestrator does not perform judgments according to the order in which the data packets arrive, but instead rearranges the timing based on a unified clock reference to eliminate timing deviations introduced by differences in network transmission paths.

[0415] Each player independently calculates and outputs the intended pulse intensity. The orchestrator calculates the overall collaboration score based on this and uploads it along with the time window metadata to the server.

[0416] The server does not directly accept the collaboration results submitted by the client, but performs a secondary feasibility check based on the current stage of the BOSS, skill cooldown status, terrain collision conditions, and team rules.

[0417] If the collaboration conditions are met, the server sends a confirmation command and phase alignment parameters. The client then fine-tunes the local phase of each character based on the parameters to achieve synchronized alignment of the three characters' animations. If the collaboration conditions are not met, the server returns a rejection reason and triggers an alternative feedback action to prevent the player from perceiving that there is no response to the input.

[0418] It is worth noting that in this embodiment, the rendering layer and the settlement layer adopt a separate architecture. In other words, the rendering layer prioritizes smoothness and continuity, while the settlement layer adheres to the principle of authoritative consistency. The two are linked through a unified action identifier, facilitating problem review and anti-fraud verification. Furthermore, the operation log simultaneously records the difference between the candidate intent and the final judgment, allowing operators to distinguish the source of the problem, including motion recognition deviations, improper rule configurations, or network jitter.

[0419] It is also worth noting that when the virtual character control method in the game provided in this application embodiment is applied to this scenario, and after two weeks of gray-scale operation of the dungeon, the success rate of the three-person cooperative skill triggering rate increases from 78.6% to 92.3%, and the proportion of failures without clear feedback in the failure scenario is significantly reduced. Among them, the cooperative score calculation method can be obtained by formula (47).

[0420] (47) In the formula, S t express t The three-person collaborative scoring at any given moment; α k Indicates the first k The intention weight of the top players; express t Time of the first k The intensity of the player's intention pulse; β This represents the phase dispersion penalty coefficient; Dt t express t Phase dispersion of the movement of the formation at any given moment. It is worth noting that, by analyzing the phase dispersion Dt t By imposing constraints, the system prioritizes selecting collaborative windows that meet both synchronization and interpretability requirements, rather than simply selecting the action moment with the highest individual human perception confidence.

[0421] The server rejection rate initially rose but gradually declined, indicating that the rule configuration and recognition model tended to align under the closed-loop log feedback. The dungeon experience survey results showed that players had a clear understanding of the reasons for cooperation failures, thanks to the system's provision of alternative action prompts and stage status feedback.

[0422] Therefore, from the perspective of R&D efficiency, the collaborative orchestrator concentrates the multi-person collaboration logic into a single module, and only needs to adjust the weight parameters and rule configurations to complete the function update during the version iteration process.

[0423] Thirdly, please refer to Figure 12 The content shown is as follows. Figure 12 This application describes the application of the virtual character control method provided in this embodiment of a dance social game. Specifically, a certain application scenario corresponds to a dance social game scenario, with the main business growth points being user-generated content (UGC) choreography and live interaction. In the traditional production process, after creators upload motion templates, they need to undergo manual quality inspection and secondary annotation, which results in problems such as long online cycles and insufficient consistency of scoring standards.

[0424] Based on this, the virtual character control method in the game provided in this application embodiment can be applied to this scenario, thereby enabling automatic cleaning of motion templates and standardization of skeletons during the production stage, and achieving real-time player follow-up scoring and adaptive difficulty adjustment during the operation stage. It is understood that this scenario, in addition to low latency requirements, also needs to consider multiple dimensions such as content production efficiency, scoring fairness, and consistency of live stream visuals.

[0425] Specifically, after the UGC creator completes the motion recording, the system first performs offline topology self-healing processing, mapping the skeletal trajectories obtained from different body types and different acquisition devices to a unified template space, eliminating the impact of differences in acquisition sources on subsequent processing.

[0426] The standardized motion trajectory is segmented into reusable motion segments. The system automatically labels each segment with attributes such as style category, difficulty level, and beat nodes, reducing the workload of manual segment-by-segment labeling.

[0427] After the action template is entered into the database, players input their intention pulse alignment module in real time. The system performs matching and judgment based on the semantic consistency of the action within the beat time window, rather than using the original coordinate point-by-point comparison method, thus improving the fairness of scoring among players of different body types.

[0428] The scoring module outputs scores and visual feedback in real time, including beat-hit effects and posture correction prompts. Feedback latency is controlled within a single beat window, ensuring a synchronized visual experience for viewers in live streaming scenarios.

[0429] The leaderboard uses a server-based authoritative scoring mechanism to prevent local data tampering. Promotional activities have tiered reward rules based on difficulty levels to prevent low-difficulty templates from being used for high-frequency score manipulation.

[0430] Online operation logs are continuously fed back to the template library, dynamically updating the scoring weights of different styles of tracks, enabling the system to adapt to newly added dance styles and movement expressions.

[0431] It is worth noting that after applying the virtual character control method in the game provided in this application embodiment, the average cycle from submission for review to official launch of the UGC template is shortened from 3.2 days to 1.1 days, indicating that this application embodiment not only supports real-time interactive scenarios, but also optimizes the operational efficiency of the content production chain. Specifically, the real-time scoring function is defined as shown in formula (48).

[0432] (48) In the formula, R t express t Real-time scoring of dance moves; c 1. c 2. c3 indicates the weighting coefficient of each rating dimension; BeatAlign t This indicates the score for beat alignment; IntentMatch t This represents the semantic matching score of the action; Stability t Indicates the score for motion stability Furthermore, data from the live streaming event showed a decrease in negative feedback from viewers regarding the asynchrony between movements and music, and a reduction in complaints from creators concerning the fairness of the scoring. As a result, the development team no longer needs to write separate judgment logic for each dance track; scoring rules and template management are all integrated into a unified, configurable framework.

[0433] To facilitate better implementation of the virtual character control method in games according to the embodiments of this application, the embodiments of this application also provide a virtual character control device in games. Please refer to... Figure 13 , Figure 13 A schematic diagram of the structure of a virtual character control device in a game provided in this application embodiment. The virtual character control device 200 in the game includes: The acquisition module 210 is used to acquire the motion input data of the target object; The object pose determination module 220 is used to determine the first skeleton pose data of the target object based on the somatosensory input data. The character pose determination module 230 is used to map the first skeleton pose data of the target object to the virtual character skeleton of the target virtual character in the target game, so as to obtain the second skeleton pose data of the target virtual character. The first action recognition module 240 is used to identify the virtual actions to be performed by the target virtual character based on the second skeleton posture data, and obtain multiple candidate virtual actions; The second action recognition module 250 is used to determine the target virtual action based on multiple candidate virtual actions, action control delay information and the motion constraints of the virtual character corresponding to the target game. The action control delay information is used to indicate the delay between the target object performing the action and the target virtual character responding to the target object's action. The control module 260 is used to control the target virtual character to perform target virtual actions.

[0434] In some embodiments provided in this application, the motion-sensing input data includes multiple sub-motion-sensing data, each of which is acquired through a terminal device. Furthermore, the object pose determination module 220 is also used to perform time alignment processing on the multiple sub-motion-sensing data to obtain multiple processed sub-motion-sensing data, and to perform fusion processing on the multiple processed sub-motion-sensing data to obtain fused motion-sensing data, and to determine the first skeleton pose data based on the fused motion-sensing data.

[0435] In some embodiments provided in this application, the object pose determination module 220 is further configured to map each sub-motion data to a target time axis to obtain mapped sub-motion data corresponding to each sub-motion data, and determine the clock drift amount of each sub-motion data relative to the target time axis based on the timestamp information of each sub-motion data and the target time axis, and perform compensation processing on the mapped sub-motion data corresponding to each sub-motion data based on the clock drift amount corresponding to each sub-motion data to obtain multiple processed sub-motion data.

[0436] In some embodiments provided in this application, the fused motion sensing data includes the observed position of each joint of the target object at the current moment. Furthermore, the object pose determination module 220 is also used to obtain the confidence level and predicted position of the observed position of each joint at the current moment, and to determine the target position of each joint at the current moment based on the observed position, the confidence level, and the predicted position, and to determine the target position of each joint at the current moment as the first skeleton pose data. The predicted position is determined by the observed position at least one moment prior to the current moment.

[0437] In some embodiments provided in this application, the virtual character control device 200 further includes a confidence level determination module. The confidence level determination module is used to determine a first confidence level component corresponding to a joint based on the tracking status of the joint points of the target object; to determine a second confidence level component corresponding to a joint based on the deviation between the observed position of the joint and the anatomical constraints of the joint; to determine a third confidence level component corresponding to the joint based on the observation consistency among multiple sensors corresponding to the joint; and to determine the observation position confidence level based on at least one of the first, second, and third confidence level components.

[0438] In some embodiments provided in this application, the object pose determination module 220 is further configured to determine the joint position anomaly score based on the deviation between the observed position and the predicted position of each joint point, and if the joint position anomaly score is less than or equal to a preset score threshold, determine the observed position weight and predicted position weight corresponding to each joint point based on the observed position confidence of each joint point, and perform weighted processing on the observed position and corresponding observed position weight of each joint point, and the predicted position and corresponding predicted position weight of each joint point to obtain the target position of each joint point at the current time.

[0439] In some embodiments provided in this application, the object pose determination module 220 is further configured to determine the target position of each joint at the current time based on the target position of each joint at the previous time when the joint position abnormal score is greater than a preset score threshold.

[0440] In some embodiments provided in this application, the character pose determination module 230 is further configured to determine the topological mapping relationship between the first joint point in the object skeleton of the target object and the second joint point in the virtual character skeleton, and, based on the topological mapping relationship, determine the target deviation data between the object skeleton and the virtual character skeleton, and, based on the topological mapping relationship, the target deviation data, and the parent-child bone segment kinematic constraint data of the virtual character skeleton, solve the pose transformation relationship between the object skeleton and the virtual character skeleton to obtain pose transformation data, and, based on the pose transformation data, transform the first skeleton pose data to obtain the second skeleton pose data of the target virtual character. The target deviation data is used to indicate at least one of the following: bone segment ratio deviation, joint rotation angle deviation, bone segment length deviation, and skeleton movement style deviation between the object skeleton and the virtual character skeleton; the parent-child bone segment kinematic constraint data is used to constrain the deviations between multiple bone segments in the virtual character skeleton that have a dependency relationship.

[0441] In some embodiments provided in this application, the first action recognition module 240 is further configured to determine the action intention of the target virtual character based on the second skeleton posture data and the game context state data of the target virtual character in the target game, and to predict the virtual action to be performed by the target virtual character based on the action intention, the second skeleton posture data and the game context state data, thereby obtaining multiple candidate virtual actions.

[0442] In some embodiments provided in this application, the virtual character control device 200 further includes a delay acquisition module and a delay determination module. The delay acquisition module is used to acquire the predicted time difference and phase difference between the time difference prediction and the response of the target virtual character to the time difference prediction. The delay determination module is used to determine the predicted time difference and phase difference as action control delay information.

[0443] In some embodiments provided in this application, the first action recognition module 240 is further configured to construct a cost function based on multiple candidate virtual actions, action control delay information, and motion constraints of the virtual character corresponding to the target game, and to solve the cost function to obtain the target virtual action.

[0444] In some embodiments provided in this application, the cost function includes at least two of the following: a posture loss term for constraining stable skeleton mapping, an intent loss term for constraining reliable intent recognition, a latency loss term for constraining controllable end-to-end latency, a rule loss term for constraining legal action gameplay, a smoothing loss term for constraining smooth action timing, and a fairness loss term for constraining controllable trigger differences among different player groups.

[0445] In some embodiments provided in this application, virtual character motion constraints include physical constraints and / or game rule constraints; physical constraints are used to limit the physical feasibility of the target virtual character's actions, and game rule constraints are used to limit the target virtual character's actions to conform to the gameplay rules of the target game.

[0446] In some embodiments provided in this application, the virtual character control device 200 further includes an action state acquisition module, a state deviation determination module, a first state transformation module, and a second state change module.

[0447] The action state acquisition module acquires the first action state of the target virtual character in the game client at the current moment, and the second action state of the target virtual character in the game server at the current moment. The state deviation determination module determines the state deviation data between the first action state and the second action state. The first state transition module controls the action state of the target virtual character in the game client to change from the first action state to the second action state at the current moment if the state deviation data is greater than or equal to a preset state deviation threshold. The second state change module controls the action state of the target virtual character in the game client to gradually change from the first action state to the second action state over multiple moments after the current moment if the state deviation data is less than the preset state deviation threshold.

[0448] The various units in the virtual character control device of the aforementioned game can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to these units.

[0449] The virtual character control device 200 in the game can be integrated into a terminal or server that has storage and a processor and thus computing power, or the virtual character control device 200 in the game can be that terminal or server.

[0450] Optionally, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0451] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a terminal or a server. Figure 14As shown, the electronic device 300 includes a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, and a computer program stored in the memory 302 and executable on the processor. The processor 301 and the memory 302 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0452] The processor 301 is the control center of the electronic device 300. It connects various parts of the electronic device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device 300 and processes data, thereby performing overall processing of the electronic device 300.

[0453] In this embodiment, the processor 301 in the electronic device 300 loads the instructions corresponding to the processes of one or more computer programs into the memory 302 according to the following steps, and the processor 301 runs the computer programs stored in the memory 302 to realize various functions: Acquire the motion input data of the target object; Based on the motion input data, determine the first skeleton pose data of the target object; The first skeleton pose data of the target object is mapped to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton pose data of the target virtual character; Based on the pose data of the second skeleton, the virtual actions to be performed by the target virtual character are identified, and multiple candidate virtual actions are obtained; Based on multiple candidate virtual actions, action control delay information, and motion constraints of the virtual character corresponding to the target game, the target virtual action is determined. The action control delay information is used to indicate the delay between the target object executing the action and the target virtual character responding to the target object's action. Control the target virtual character to perform the target virtual actions.

[0454] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0455] Optional, such as Figure 14 As shown, the electronic device 300 also includes: a display screen 303, a radio frequency circuit 304, an audio circuit 305, an input unit 306, and a power supply 307. The processor 301 is electrically connected to the display screen 303, the radio frequency circuit 304, the audio circuit 305, the input unit 306, and the power supply 307. Those skilled in the art will understand that... Figure 14 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0456] The display screen 303 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The display screen 303 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. Optionally, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 301, and can receive and execute commands from the processor 301. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 301 to determine the type of touch event. Subsequently, the processor 301 provides corresponding visual output on the display panel according to the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the display screen 303 to achieve input and output functions. However, in some embodiments, the touch panel and the display screen 303 can be implemented as two independent components to achieve input and output functions. That is, the display screen 303 can also be used as part of the input unit 306 to achieve input functions.

[0457] The radio frequency circuit 304 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0458] Audio circuitry 305 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuitry 305 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 305, converted back into audio data, and then processed by processor 301 before being transmitted via radio frequency circuitry 304 to, for example, another electronic device, or output to memory 302 for further processing. Audio circuitry 305 may also include an earphone jack to facilitate communication between peripheral headphones and electronic devices.

[0459] The input unit 306 can be used to receive input numbers, characters, or object feature information (such as fingerprints, irises, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0460] Power supply 307 is used to supply power to various components of electronic device 300. Optionally, power supply 307 can be logically connected to processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 307 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0461] although Figure 14 As not shown in the diagram, the electronic device 300 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0462] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding flow in the virtual character control method in the game described in this application embodiment; for brevity, further details are omitted here.

[0463] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding flow in the virtual character control method in the game described in this application embodiment. For simplicity, further details are omitted here.

[0464] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding flow in the virtual character control method in the game described in this application. For brevity, further details are omitted here.

[0465] It should be understood that the processor in this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0466] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0467] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0468] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0469] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0470] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0471] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0472] In addition, the functional units in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0473] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0474] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for controlling a virtual character in a game, characterized in that, The method includes: Acquire the motion input data of the target object; Based on the somatosensory input data, the first skeleton pose data of the target object is determined; The first skeleton pose data of the target object is mapped to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton pose data of the target virtual character. Based on the second skeleton pose data, the virtual actions to be performed by the target virtual character are identified, and multiple candidate virtual actions are obtained; Based on the multiple candidate virtual actions, action control delay information, and the motion constraints of the virtual character corresponding to the target game, a target virtual action is determined, wherein the action control delay information is used to indicate the delay between the target object performing the action and the target virtual character responding to the target object performing the action; Control the target virtual character to perform the target virtual action.

2. The method according to claim 1, characterized in that, The somatosensory input data includes multiple sub-spontaneous data, and each sub-spontaneous data is acquired through a terminal device. Determining the first skeleton pose data of the target object based on the somatosensory input data includes: The multiple sub-sensory data are time-aligned to obtain multiple processed sub-sensory data. The multiple processed somatosensory data are fused to obtain fused somatosensory data; Based on the fused somatosensory data, the first skeleton posture data is determined.

3. The method according to claim 2, characterized in that, The process of performing time alignment on the multiple sub-sensory data results in multiple processed sub-sensory data, including: Each of the sub-sensory data points is mapped to the target time axis to obtain the mapped sub-sensory data corresponding to each sub-sensory data point. Based on the timestamp information of each sub-sensory data and the target time axis, determine the clock drift amount of each sub-sensory data relative to the target time axis; Based on the clock drift amount corresponding to each of the sub-sensory data, compensation processing is performed on the mapped sub-sensory data corresponding to each of the sub-sensory data to obtain the plurality of processed sub-sensory data.

4. The method according to claim 2, characterized in that, The fused somatosensory data includes the observed position of each joint of the target object at the current moment; The step of determining the first skeleton pose data based on the fused somatosensory data includes: Obtain the observed position confidence and predicted position of each of the key points at the current time, wherein the predicted position is determined by the observed position at least one time before the current time; Based on the observed position, the confidence level of the observed position, and the predicted position of each of the key points, the target position of each key point at the current time is determined; The target position of each joint at the current moment is determined as the first skeleton posture data.

5. The method according to claim 4, characterized in that, The confidence level of the observation location is determined in the following way: Based on the tracking status of the joints of the target object, determine the first confidence component corresponding to the joint; The second confidence component corresponding to the joint is determined based on the deviation between the observed position of the joint and the anatomical constraints of the joint. The third confidence component corresponding to the key point is determined based on the observation consistency among multiple sensors corresponding to the key point. The confidence level of the observation location is determined based on at least one of the first confidence component, the second confidence component, and the third confidence component.

6. The method according to claim 4, characterized in that, Determining the target position of each joint at the current time based on the observed position, the confidence level of the observed position, and the predicted position of each joint includes: A joint position anomaly score is determined based on the deviation between the observed position and the predicted position of each joint point; If the joint position anomaly score is less than or equal to a preset score threshold, the observation position weight and predicted position weight corresponding to each joint point are determined based on the observation position confidence of each joint point. The observed position and corresponding observed position weight of each joint point, and the predicted position and corresponding predicted position weight of each joint point are weighted to obtain the target position of each joint point at the current time.

7. The method according to claim 6, characterized in that, The step of determining the target position of each joint at the current time based on the observed position, the confidence level of the observed position, and the predicted position of each joint further includes: If the joint position abnormality score is greater than a preset score threshold, the target position of each joint at the current time is determined based on the target position of each joint at the previous time.

8. The method according to claim 1, characterized in that, The step of mapping the first skeleton pose data of the target object to the virtual character skeleton of the target virtual character in the target game to obtain the second skeleton pose data of the target virtual character includes: Determine the topological mapping relationship between the first joint point in the object skeleton of the target object and the second joint point in the virtual character skeleton; Based on the topological mapping relationship, target deviation data between the object skeleton and the virtual character skeleton is determined, wherein the target deviation data is used to indicate at least one of the following: bone segment ratio deviation, joint rotation angle deviation, bone segment length deviation, and skeleton movement style deviation between the object skeleton and the virtual character skeleton. Based on the topological mapping relationship, the target deviation data, and the parent-child bone segment kinematic constraint data of the virtual character skeleton, the posture transformation relationship between the object skeleton and the virtual character skeleton is solved to obtain posture transformation data. The parent-child bone segment kinematic constraint data is used to constrain the deviation between multiple bone segments with dependent relationships in the virtual character skeleton. Based on the posture transformation data, the first skeleton posture data is transformed to obtain the second skeleton posture data of the target virtual character.

9. The method according to claim 1, characterized in that, Based on the second skeleton pose data, the virtual actions to be performed by the target virtual character are identified, resulting in multiple candidate virtual actions, including: Based on the second skeleton posture data and the game context state data of the target virtual character in the target game, the action intention of the target virtual character is determined; Based on the action intent, the second skeleton posture data, and the game context state data, the virtual action to be performed by the target virtual character is predicted to obtain the multiple candidate virtual actions.

10. The method according to claim 1, characterized in that, The method further includes: Predict the time difference and phase difference between the target object performing an action and the target virtual character responding to the target object's action. The time difference prediction and the phase difference prediction are determined as the motion control delay information.

11. The method according to claim 1, characterized in that, The process of determining the target virtual action based on the multiple candidate virtual actions, motion control delay information, and the motion constraints of the virtual character corresponding to the target game includes: A cost function is constructed based on the multiple candidate virtual actions, the action control delay information, and the motion constraints of the virtual character corresponding to the target game. The cost function is solved to obtain the target virtual action.

12. The method according to claim 11, characterized in that, The cost function includes at least two of the following: The following loss terms are used to constrain the stability of the skeleton mapping: the intention loss term to constrain the reliability of intention recognition; the latency loss term to constrain the controllable end-to-end latency; the rule loss term to constrain the legality of the action gameplay; the smoothing loss term to constrain the smoothness of the action timing; and the fairness loss term to constrain the controllable differences in triggering between different player groups.

13. The method according to claim 1 or 11, characterized in that, The virtual character motion constraints include physical constraints and / or game rule constraints; the physical constraints are used to limit the physical feasibility of the target virtual character's actions, and the game rule constraints are used to limit the target virtual character's actions to conform to the gameplay rules of the target game.

14. The method according to claim 1, characterized in that, The method further includes: Obtain the first action state of the target virtual character in the game client of the target game at the current moment, and the second action state of the target virtual character in the game server of the target game at the current moment; Determine the state deviation data between the first action state and the second action state; If the state deviation data is greater than or equal to a preset state deviation threshold, the action state of the target virtual character in the game client is controlled to change from the first action state to the second action state at the current moment. If the state deviation data is less than the preset state deviation threshold, the action state of the target virtual character in the game client is controlled to gradually change from the first action state to the second action state over multiple time intervals after the current time.

15. A virtual character control device for a game, characterized in that, The device includes: The acquisition module is used to acquire the motion input data of the target object; An object pose determination module is used to determine the first skeleton pose data of the target object based on the somatosensory input data. The character pose determination module is used to map the first skeleton pose data of the target object to the virtual character skeleton of the target virtual character in the target game, so as to obtain the second skeleton pose data of the target virtual character. The first action recognition module is used to identify the virtual action to be performed by the target virtual character based on the second skeleton posture data, and obtain multiple candidate virtual actions; The second action recognition module is used to determine the target virtual action based on the multiple candidate virtual actions, action control delay information and the motion constraints of the virtual character corresponding to the target game, wherein the action control delay information is used to indicate the delay between the target object performing the action and the target virtual character responding to the target object performing the action; The control module is used to control the target virtual character to perform the target virtual action.

Images