A full-cycle motion tracking method and system based on multi-modal data mining

Abstract

The application belongs to the field of motion tracking, and discloses a full-cycle motion tracking method and system based on multi-modal data mining. The application obtains multi-modal health data of a user in a preset evaluation period, constructs a health evaluation index matrix to calculate a dynamic weight and a health comprehensive index, can unify and fuse motion data, physiological morphology data and subjective feedback data, and overcomes the defects of single evaluation dimension and one-sided results of the prior art. The application calculates a motion energy value based on a motion segment, an intensity coefficient, an effective motion duration and a health change gain, can establish a correlation between motion input and health improvement results, and improves the accuracy of incentive feedback. The application maps the motion energy value to virtual pet growth resources, drives experience accumulation, grade promotion, appearance unlocking and action slot expansion, and can enhance the sustainability of the incentive mechanism.

Description

Technical Field

[0001] This invention belongs to the field of motion tracking, specifically relating to a full-cycle motion tracking method and system based on multimodal data mining. Background Technology

[0002] With the development of mobile internet, smart wearable devices, augmented reality (AR) technology, and artificial intelligence (AI) technology, digital systems for public fitness and sports health management are becoming increasingly widespread. Existing sports and health platforms typically collect basic exercise data such as steps, exercise duration, heart rate, and calorie consumption through mobile terminals or wearable devices. Based on preset rules, they provide users with functions such as course recommendations, exercise records, reminders for progress goals, and social sharing. Some systems also introduce virtual avatars, badges, leaderboards, or location-based exploration features to enhance user engagement. Furthermore, some AR sports systems can provide demonstrations of standard movements or simple virtual interactions during exercise to increase user enjoyment.

[0003] However, current technologies primarily focus on data processing within a single dimension of movement behavior, often lacking collaborative analysis of multimodal health information such as sleep quality, body shape, physiological state, and subjective fatigue. This results in somewhat one-sided health assessments that fail to accurately reflect the user's overall health trends. Furthermore, existing incentive methods largely rely on fixed points, badge rewards, or leaderboard competition, failing to deeply integrate real-world health improvements with a long-term, growth-oriented virtual platform, leading to weak long-term user engagement. Additionally, current augmented reality interactions are mostly static displays or pre-set action guidance, lacking dynamic, companion-style interaction capabilities based on the user's real-time movement status, making it difficult to create an immersive and continuous exercise experience. Moreover, current exercise and dietary recommendations are typically generated based on generic templates, lacking a deep, personalized adjustment mechanism that combines historical health trends, real-time assessment results, and individual preferences. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of insufficient multimodal health data fusion, poor dynamics of health assessment, lack of continuous personalized binding of incentive mechanisms, and insufficient real-time companionship and interaction capabilities in existing sports and health systems, and to provide a full-cycle sports tracking method and system based on multimodal data mining.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a full-cycle motion tracking method based on multimodal data mining, comprising the following steps: Acquire multimodal health data of users within a preset assessment period; The multimodal health data is standardized to construct a health assessment index matrix, and the dynamic weight of each health assessment index in the current assessment period is calculated based on the health assessment index matrix. Based on dynamic weights, the health assessment indicator matrix is ​​weighted and the user's comprehensive health index for the current assessment period is calculated. The system acquires data from each segment of the user's current exercise process, calculates the baseline exercise value based on the intensity coefficient and effective exercise duration of each segment, and calculates the corresponding exercise energy value by combining the gain of the comprehensive health index relative to the historical periodic moving average. The exercise energy value is mapped to the virtual pet growth resource, and the virtual pet growth resource drives the accumulation of virtual pet experience, level increase, appearance unlocking and action slot expansion. The system acquires the user's real-time displacement vector, movement speed, and real-time movement intensity information to drive the virtual pet in the augmented reality scene to perform synchronous follow actions and status feedback interactions corresponding to the user's movement state. A digital twin of a user's health is constructed based on the historical change sequence of the comprehensive health index, and personalized exercise and dietary adjustment suggestions are output by combining the retrieval-enhanced generation model.

[0006] A further improvement of this invention is that the multimodal health data includes exercise data, physiological morphology data, and subjective feedback data; wherein, the exercise data includes exercise type, exercise duration, exercise intensity, exercise trajectory, and estimated calorie consumption; the physiological morphology data includes height, weight, body fat percentage, resting heart rate, and sleep data; and the subjective feedback data includes fatigue score and exercise mood score.

[0007] A further improvement of this invention lies in the following method for calculating the dynamic weight of each health assessment indicator within the current assessment period based on the health assessment indicator matrix: Obtain the health assessment indicator matrix, and perform missing value completion and outlier removal on each indicator in the health assessment indicator matrix; Based on preset standardization rules, dimensionless normalization is performed on each indicator; Calculate the corresponding information entropy value for each normalized indicator; Calculate the difference coefficient of the corresponding indicator based on the entropy value of each information; The dynamic weight of each health assessment indicator in the current assessment period is determined based on the proportion of each difference coefficient to the total of all difference coefficients.

[0008] A further improvement of this invention lies in the following method for calculating the user's comprehensive health index within the current assessment period: Obtain the weighted health assessment index matrix, and then weight each index in the health assessment index matrix according to dynamic weights to generate a weighted evaluation matrix. The maximum value of each indicator in the weighted evaluation matrix is ​​extracted as the positive ideal solution, and the minimum value is extracted as the negative ideal solution. Calculate the first distance between the user's current evaluation data and the ideal solution; Calculate the second distance between the user's current evaluation data and the negative ideal solution; The comprehensive health index is determined based on the ratio of the second distance to the sum of the first distance and the second distance.

[0009] A further improvement of this invention lies in the following method for calculating the corresponding exercise energy value by combining the gain of change in the comprehensive health index relative to the historical periodic moving average: Obtain the change gain of the historical periodic moving average, and divide the user's current motion process into motion segments according to the continuous motion duration change and motion state switching nodes; Based on the exercise type and real-time heart rate zone corresponding to each exercise segment, the intensity coefficient is matched accordingly. The intensity coefficient of each exercise segment is multiplied by the corresponding effective exercise duration and then summed to obtain the baseline exercise value. The gain on health change is calculated based on the comprehensive health index of the current assessment period and the moving average within a preset historical period window. The exercise energy value corresponding to the current exercise is generated based on the product of the baseline exercise value and the health change gain.

[0010] A further improvement of this invention lies in the following method for obtaining the user's real-time displacement vector, movement speed, and real-time movement intensity information, and driving the virtual pet in the augmented reality scene to perform synchronized following actions and state feedback interactions corresponding to the user's movement state: Acquire real-time displacement vector, motion velocity, and real-time motion intensity information of the user, and identify target planes in the real environment based on images from the mobile terminal's camera. Establish augmented reality anchor point coordinates on the target plane; The virtual pet's position relative to the augmented reality anchor point is updated based on the user's real-time displacement vector. Match the virtual pet's walking, running, or jumping animations to the user's movement speed; The virtual pet's facial expressions and voice feedback are switched according to the user's real-time exercise intensity.

[0011] A further improvement of this invention is that, when a user submits a transaction request, the item identifier, validity period status, and listing energy value of the virtual item to be traded are written into the transaction queue. Transaction matching is performed based on item type, listed energy value, and user purchase conditions; After a successful match, the corresponding exercise energy value will be deducted from the buyer's account and written to the seller's account; The ownership identifiers of traded items are updated synchronously.

[0012] Secondly, the present invention provides a full-cycle motion tracking system based on multimodal data mining, comprising: The data acquisition module is used to acquire the user's multimodal health data within a preset assessment period; The dynamic weighting module is used to standardize multimodal health data, construct a health assessment indicator matrix, and calculate the dynamic weight of each health assessment indicator in the current assessment period based on the health assessment indicator matrix. The comprehensive index module is used to perform weighted processing on the health assessment indicator matrix based on dynamic weights and calculate the user's comprehensive health index within the current assessment period. The exercise energy module is used to acquire data of each exercise segment during the user's current exercise process, calculate the basic exercise value based on the intensity coefficient and effective exercise duration of each exercise segment, and calculate the corresponding exercise energy value by combining the change gain of the comprehensive health index relative to the historical periodic moving average. The pet enhancement module maps exercise energy values ​​to virtual pet growth resources, which drive the accumulation of virtual pet experience, level increase, appearance unlocking, and action slot expansion based on virtual pet growth resources; The interaction module is used to obtain the user's real-time displacement vector, movement speed and real-time movement intensity information, and drive the virtual pet in the augmented reality scene to perform synchronous following actions and status feedback interactions corresponding to the user's movement state; The output module is used to construct a digital twin of the user's health based on the historical change sequence of the comprehensive health index, and output personalized exercise suggestions and dietary adjustment suggestions in combination with the retrieval enhancement generation model.

[0013] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a full-cycle motion tracking method based on multimodal data mining.

[0014] Fourthly, the present invention provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a full-cycle motion tracking method based on multimodal data mining.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention acquires multimodal health data from users within a preset assessment period and constructs a health assessment index matrix to calculate dynamic weights and a comprehensive health index. This unified integration of exercise data, physiological morphology data, and subjective feedback data overcomes the shortcomings of existing technologies, such as single assessment dimensions and one-sided results. Furthermore, by calculating exercise energy values ​​based on exercise segments, intensity coefficients, effective exercise duration, and health change gains, this invention establishes a correlation between exercise input and health improvement results, enhancing the accuracy of incentive feedback. By mapping exercise energy values ​​to virtual pet growth resources, driving experience accumulation, level increases, appearance unlocking, and action slot expansion, this invention enhances the sustainability of the incentive mechanism. By driving virtual pets in augmented reality scenarios to perform synchronized follow-up actions and status feedback interactions, this invention improves interactivity and immersion during exercise. Finally, by constructing a user's health digital twin and combining it with a retrieval-enhanced generation model to output personalized exercise and dietary adjustment suggestions, this invention improves the adaptability of the suggestions to the user's current health status, thus forming a full-cycle, closed-loop exercise tracking and health promotion solution. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention; Figure 2 This is a system diagram of the present invention; Figure 3 A schematic diagram of the interface for an electronic pet growth and decoration system; Figure 4 A schematic diagram of a real-time AR pet training scenario; Figure 5 This is a system diagram for an embodiment. Detailed Implementation

[0017] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0018] Example 1: See Figure 1 A full-cycle motion tracking method based on multimodal data mining includes the following steps: S1, acquire the user's multimodal health data within a preset assessment period.

[0019] S2 standardizes multimodal health data, constructs a health assessment index matrix, and calculates the dynamic weight of each health assessment index in the current assessment period based on the health assessment index matrix.

[0020] S3, based on dynamic weights, performs weighted processing on the health assessment indicator matrix and calculates the user's comprehensive health index within the current assessment period.

[0021] S4: Obtain data of each exercise segment during the user's current exercise process, calculate the basic exercise value based on the intensity coefficient and effective exercise duration of each exercise segment, and calculate the corresponding exercise energy value by combining the change gain of the comprehensive health index relative to the historical periodic moving average.

[0022] S5 maps exercise energy values ​​to virtual pet growth resources, and drives the accumulation of virtual pet experience, level increase, appearance unlocking and action slot expansion based on virtual pet growth resources.

[0023] S6 acquires the user's real-time displacement vector, movement speed, and real-time movement intensity information, and drives the virtual pet in the augmented reality scene to perform synchronized following actions and status feedback interactions corresponding to the user's movement state.

[0024] S7 constructs a digital twin of the user's health based on the historical change sequence of the comprehensive health index, and outputs personalized exercise suggestions and dietary adjustment suggestions by combining the retrieval enhancement generation model.

[0025] Example 2: See Figure 2 A full-cycle motion tracking system based on multimodal data mining, comprising: The data acquisition module is used to acquire the user's multimodal health data within a preset assessment period.

[0026] The dynamic weighting module is used to standardize multimodal health data, construct a health assessment indicator matrix, and calculate the dynamic weight of each health assessment indicator in the current assessment period based on the health assessment indicator matrix.

[0027] The comprehensive index module is used to perform weighted processing on the health assessment indicator matrix based on dynamic weights and calculate the user's comprehensive health index within the current assessment period.

[0028] The exercise energy module is used to acquire data from each exercise segment during the user's current exercise process. Based on the intensity coefficient and effective exercise duration of each exercise segment, it calculates the basic exercise value and combines the gain of the comprehensive health index relative to the historical periodic moving average to calculate the corresponding exercise energy value.

[0029] The pet enhancement module maps exercise energy values ​​to virtual pet growth resources, which drive the accumulation of virtual pet experience, level increase, appearance unlocking, and action slot expansion.

[0030] The interaction module is used to obtain the user's real-time displacement vector, movement speed, and real-time movement intensity information, and drive the virtual pet in the augmented reality scene to perform synchronous following actions and status feedback interactions corresponding to the user's movement state.

[0031] The output module is used to construct a digital twin of the user's health based on the historical change sequence of the comprehensive health index, and output personalized exercise suggestions and dietary adjustment suggestions in combination with the retrieval enhancement generation model.

[0032] Example 3: This embodiment is used to explain in detail the specific implementation of multimodal health data, including exercise data, physiological morphology data, and subjective feedback data.

[0033] First, after a user completes registration, a personal health profile is created. This profile pre-stores the user's identity, age, gender, height, initial weight, basic training goals, common exercise types, and device binding information. Upon entering a preset assessment cycle, the system collects three types of data through mobile terminal sensors, external device interfaces, and the human-computer interaction interface. The first type is exercise data, which includes at least exercise type, exercise duration, exercise intensity, exercise trajectory, and estimated calorie consumption. The exercise type can be manually selected by the user or automatically identified by the system based on acceleration change patterns, GPS speed curves, and heart rate variation patterns. Exercise duration is calculated using start and end timestamps, and the intermediate process is recorded at the second or 5-second level. Exercise intensity is obtained by jointly mapping real-time heart rate zones with exercise type; for example, the maximum heart rate percentage is divided into low-intensity, medium-intensity, and high-intensity zones, and intensity labels are output based on types such as running, walking, and cycling. The exercise trajectory is continuously collected via the GPS module to form a set of pathpoints, and drift points are filtered out. The estimated calorie consumption is calculated based on a combination of weight, metabolic equivalent, heart rate, and exercise duration. The second category is physiological data, including at least height, weight, body fat percentage, resting heart rate, and sleep data. Height can be provided from the initial profile; weight and body fat percentage can be obtained synchronously via a body fat scale or manual input by the user; resting heart rate is preferably the average value under quiet conditions at night or in the early morning; sleep data includes at least one of the following: total sleep duration, deep sleep duration, light sleep duration, REM sleep duration, number of nighttime awakenings, and PSQI score. The third category is subjective feedback data, including fatigue ratings and exercise mood ratings, preferably entered by the user in the interface after daily exercise or before bedtime. To ensure direct access to subsequent health assessments, the system performs uniform unit conversions on data from different sources, such as standardizing duration to minutes, heart rate to beats per minute, weight to kilograms, and trajectory to latitude and longitude time series, and attaching a uniform timestamp, source identifier, and reliability identifier to all data. Finally, the system aggregates the above three types of data according to the assessment cycle and writes them into the health data pool, providing complete input for the subsequent construction of a health assessment indicator matrix. This embodiment allows users' exercise behavior, physiological state, and subjective feelings to be organized into a unified data framework, thereby laying a data foundation for subsequent dynamic health assessments. Example 4: This embodiment details the process of standardizing multimodal health data, constructing a health assessment index matrix, and calculating the dynamic weights of each health assessment index within the current assessment period based on the health assessment index matrix.

[0034] Multiple assessment indicators were extracted from the health data pool, with a preferred selection including exercise duration, average exercise intensity, estimated calorie expenditure, BMI, body fat percentage, resting heart rate, total sleep duration, deep sleep percentage, PSQI score, fatigue score, and exercise mood score, forming a health assessment indicator matrix:

[0035] Where n represents the number of evaluation samples in the current evaluation window, and m represents the number of indicators.

[0036] To improve data quality, missing value completion and outlier removal are performed on each indicator in the matrix. Missing value completion preferably uses the moving average of the user's last 7 evaluation periods. If historical samples are insufficient, the value of the previous valid period or the group average is used for filling. Outlier removal preferably uses a combination of business threshold method and 3σ principle. For example, when the resting heart rate exceeds the reasonable physiological range or a trajectory point experiences an unreasonable long-distance jump in a very short period of time, it is marked as an outlier and removed.

[0037] A label table is generated based on the indicator type, categorizing each indicator into benefit-type indicators, cost-type indicators, and interval-optimal indicators. For benefit-type indicators, a formula is used... Perform normalization; For cost-related indicators, the formula is used. Perform normalization; For interval-optimal indices, it is preferable to first convert them into deviations from the center value of the target interval, and then perform normalization.

[0038] After obtaining the normalized matrix X', calculate the proportion of the j-th indicator in the i-th sample. And according to the formula Calculate the information entropy of each indicator, where .when season To maintain computational continuity.

[0039] Furthermore, according to the formula Calculate the difference coefficient, and then use the formula Obtain the dynamic weight vector W. Write this weight vector into the periodic weight table and record the weight change trajectory of the most recent N periods to observe the impact of different health indicators on different stages.

[0040] This embodiment allows health assessment weights to change in real time according to data distribution, rather than relying on fixed manual settings, thereby improving the objectivity, adaptability, and cross-user compatibility of health assessments. Example 5: This embodiment details the specific implementation of weighting the health assessment indicator matrix based on dynamic weights and calculating the user's comprehensive health index within the current assessment period.

[0041] Receive the normalized matrix X' and the dynamic weight vector W, perform item-by-item weighting processing on each indicator, and generate the weighted evaluation matrix V, the calculation formula of which is: .

[0042] After obtaining the weighted evaluation matrix, the maximum value is extracted by column as the positive ideal solution A+, and the minimum value is extracted by column as the negative ideal solution A-, specifically expressed as follows: as well as Here, a positive ideal solution represents a theoretical reference point that is in the optimal state across all health indicator dimensions, while a negative ideal solution represents a theoretical reference point that is in the worst state across all health indicator dimensions.

[0043] For the target sample of the current user in the current evaluation period, calculate the first distance between it and the positive ideal solution and the second distance between it and the negative ideal solution, preferably using Euclidean distance. The formulas are as follows: and .

[0044] In another alternative implementation, weighted Minkowski distance or Mahalanobis distance can be used as alternative calculation methods. Based on the above two distances, the system then calculates according to the formula... The Health Comprehensive Index (HCI) is calculated, with values ​​ranging from 0 to 1. A higher HCI value indicates that the current health status is closer to the ideal solution. To enhance subsequent analysis capabilities, the system generates a ranking of the contributions of each indicator to the current HCI, records the top K most influential indicators, and associates the HCI results with date labels, current training target labels, and current stage labels, writing them into a historical health sequence list. This historical sequence list is used both for calculating health change gains in the next embodiment and for constructing a digital twin.

[0045] This embodiment can compress multidimensional heterogeneous health data into a unified, continuous, and comparable comprehensive health indicator, enabling the system to more intuitively track the changing trends of users' health status and provide a unified input for subsequent incentive mechanisms and personalized suggestion modules.

[0046] Example 6: See Figure 3This embodiment details the specific implementation of acquiring data from each exercise segment during a user's current exercise process, calculating the baseline exercise value based on the intensity coefficient and effective exercise duration corresponding to each exercise segment, and calculating the corresponding exercise energy value by combining the gain of the comprehensive health index relative to the historical periodic moving average.

[0047] When the user begins exercising, a continuous sensor stream is initiated to collect acceleration, GPS displacement, real-time heart rate, and exercise status tags. The entire exercise process is automatically segmented based on changes in continuous exercise duration, exercise status transition points, heart rate interval thresholds, and GPS speed change rate. Each exercise segment records at least the start time, end time, average heart rate, peak heart rate, average speed, segment distance, exercise type, and effective exercise duration. .

[0048] The preset intensity coefficient mapping table is invoked to match the intensity coefficient based on the exercise type and real-time heart rate zone corresponding to the segment. For example, a lower intensity coefficient can be set for walking segments in the low heart rate zone, and a higher intensity coefficient can be set for running segments in the medium to high heart rate zone; furthermore, the intensity coefficient can be adjusted based on the user's age, gender, training stage, and personalized goals. After obtaining the intensity coefficient for each segment, it is then calculated using the formula... Calculate the baseline value BV of the current motion process.

[0049] Extracting historical cycle windows from historical health sequence lists Calculate the moving average of the HCI sequence within the range. And according to the formula Calculate the current periodic health change gain G.

[0050] In a preferred embodiment, the system can also superimpose a BMI improvement coefficient, a sleep recovery coefficient, or a fatigue reduction coefficient on top of G, so that the change gain not only reflects the improvement of the overall health index, but also reflects the degree of physical recovery and body shape improvement.

[0051] According to the formula Generate the kinetic energy value EP corresponding to the current motion, where This is the gain adjustment coefficient, which can be adjusted according to the user's growth stage or incentive strategy. After EP is generated, the system synchronously writes it into the pet growth resource pool, the user's virtual asset account, the incentive log, and the available balance table for transactions, providing direct input for subsequent pet growth and trading modules.

[0052] This embodiment enables the unified mapping of users' exercise input and health improvement results into quantifiable incentive resources, establishing a one-to-one correspondence between exercise behavior and health changes, thereby significantly improving the timeliness and personalization of incentive feedback.

[0053] Example 7: This embodiment is used to explain in detail the technical features of virtual transactions, and at the same time, to explain how exercise energy values ​​are mapped to virtual pet growth resources.

[0054] First, the exercise energy value (EP) is read and converted into pet experience points according to a preset ratio. A 1:1 mapping relationship is preferred, that is, 1 point of EP corresponds to 1 point of pet experience points. In another implementation, a segmented mapping rule can be used based on the pet's level stage, for example, the lower level stage grows faster and the higher level stage grows slower.

[0055] After experience points are written to the pet's experience pool, the system determines whether to trigger an upgrade based on the experience threshold corresponding to the current pet level. When the accumulated experience in the pool reaches the threshold for the next level, the pet's level is automatically increased, and basic appearance resources such as fur color, eye color, texture, clothing, or special effects are unlocked from the appearance resource library based on the new level. Simultaneously, new action slots are opened from the action resource library. For limited-time decorations, the system records an expiration date field in the resource object and automatically switches to an invalid state upon expiration. To build a sustainable internal economic ecosystem, a virtual trading module is further implemented.

[0056] When a user submits a transaction request, the system reads the user's current inventory resources, extracts the item identifier, item type, validity period status, listing EP value, scarcity tag, listing duration, transaction fee rate, and seller account identifier of the item to be traded, and encapsulates them into a transaction order object and writes it to the matching queue.

[0057] The system can automatically match buyer purchase requests with seller orders using one or more strategies, including price priority, time priority, and tier matching priority. Upon successful matching, the system executes account settlement logic: first, it deducts the listed EP value and transaction fee from the buyer's EP account; then, it writes the transaction fee to the system's economic pool and writes the remaining EP to the seller's account; finally, it synchronously updates the transaction log, asset snapshots for both buyers and sellers, item ownership identifiers, and inventory index table.

[0058] If the item being traded is a wearable pet accessory, the pet appearance configuration table will be further updated so that the buyer can immediately use the item in the pet interface.

[0059] Through this embodiment, a stable internal economic cycle can be formed in the chain of obtaining EP through exercise, pet growth, resource unlocking, and resource trading, so that the system incentive is no longer limited to a single point feedback, but has multiple driving forces such as growth, collection, exchange and continuous participation.

[0060] Example 8: See Figure 4This embodiment details the specific implementation method of obtaining the user's real-time displacement vector, movement speed, and real-time movement intensity information, and driving the virtual pet in the augmented reality scene to perform synchronous following actions and status feedback interactions corresponding to the user's movement state.

[0061] The system continuously captures video streams using the mobile terminal's camera and combines visual inertial odometry, plane detection algorithms, and SLAM mapping algorithms to identify target planes in the real environment. These target planes can be the ground, a running track, an indoor floor, or a tabletop. Once the target plane is successfully identified, the system establishes augmented reality anchor point coordinates at the center of the plane or a preset position in front of the user, and initially binds the 3D virtual pet model to that anchor point.

[0062] It continuously receives real-time displacement vectors and motion speed data calculated by the user using GPS, accelerometers, and gyroscopes, and updates the pet's position relative to the anchor point according to preset following rules. For example, when the user moves forward steadily, the pet can maintain a certain distance in front of the user; when the user decelerates, the pet automatically reduces its forward speed and shortens its lead distance; when the user turns, the pet performs a smooth turn according to the trajectory curvature.

[0063] To ensure that the pet's actions are consistent with the user's state, an animation state machine is set up, with actions such as walking, running, jumping, waiting, celebrating, reminding to slow down, and prompting to drink water as state nodes, and the state switches according to the user's speed range, real-time exercise intensity, and goal achievement.

[0064] The system utilizes the user's real-time heart rate or exercise intensity level to access the facial expression and voice feedback resource libraries. For example, it displays a relaxed expression during low-intensity phases and an effort or fatigue expression during high-intensity phases, while playing voice prompts such as "Keep going," "Pay attention to your breathing," and "You've reached your goal for this phase." To enhance realism, the system can also adjust the pet's shadows and posture based on lighting, occlusion, and ground normals.

[0065] This embodiment allows virtual pets to move beyond being static displays and become dynamic training partners that change in sync with the user's actual exercise, thereby significantly enhancing the immersion, interactivity, and companionship during the exercise process.

[0066] Example 9: This embodiment details how to construct a user's health digital twin based on the historical change sequence of the comprehensive health index, and how to combine it with a retrieval-enhanced generation model to output personalized exercise and dietary adjustment suggestions.

[0067] First, HCI sequences and corresponding raw indicator details for multiple assessment periods are extracted from the historical health sequence list, and feature engineering is performed to form the input features of the digital twin model. These features include at least HCI trend features, fluctuation features, periodic features, BMI change features, sleep quality change features, training frequency features, exercise type distribution features, and subjective fatigue bias features. Trend features can be obtained from linear fitting slope or moving average difference; fluctuation features can be represented by variance, standard deviation, or coefficient of variation; and periodic features can be extracted through Fourier decomposition or periodic window statistics.

[0068] After obtaining the above characteristics, it is preferable to use a random forest model to build a digital twin of the user's health, so as to output the health trend prediction results, potential risk labels and suggested target ranges within a preset period in the future; in other optional implementations, XGBoost, LSTM or temporal Transformer can also be used instead.

[0069] After completing trend prediction, the system activates the retrieval enhancement generation module. Specifically, the system constructs a semantic retrieval vector based on the user's current digital twin state and performs a dual-channel retrieval from a vectorized professional sports and health knowledge base: the first channel prioritizes the recall of knowledge fragments related to exercise prescriptions, training intensity adjustments, and recovery cycles, while the second channel prioritizes the recall of knowledge fragments related to dietary structure, energy supplementation, and sleep recovery. Subsequently, the results from the two channels are fused and reordered to select the target knowledge fragments most relevant to the current user state.

[0070] The system inputs target knowledge fragments, current health status, future trend predictions, and user preference information into a large language model. The large language model then outputs personalized exercise prescriptions, dietary adjustment suggestions, next-day recovery suggestions, and stage goal adjustment suggestions, and continuously interacts with the user through a conversational interface.

[0071] This embodiment can elevate the assessment of historical health status from describing the past to predicting the future and providing actionable solutions, thereby forming a complete health promotion closed loop from data collection, health assessment, behavioral incentives, AR interaction to intelligent suggestions.

[0072] Example 10: Please see Figure 5 As shown, the present invention also provides an electronic device 100 based on a full-cycle motion tracking method using multimodal data mining; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0073] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the full-cycle motion tracking method based on multimodal data mining described in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0074] The at least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor. The processor 102 is the control center of the electronic device 100, connecting various parts of the electronic device 100 via various interfaces and lines.

[0075] The memory 101 in the electronic device 100 stores multiple instructions to implement a full-cycle motion tracking method based on multimodal data mining, and the processor 102 can execute the multiple instructions to achieve the following: Acquire multimodal health data of users within a preset assessment period; The multimodal health data is standardized to construct a health assessment index matrix, and the dynamic weight of each health assessment index in the current assessment period is calculated based on the health assessment index matrix. Based on dynamic weights, the health assessment indicator matrix is ​​weighted and the user's comprehensive health index for the current assessment period is calculated. The system acquires data from each segment of the user's current exercise process, calculates the baseline exercise value based on the intensity coefficient and effective exercise duration of each segment, and calculates the corresponding exercise energy value by combining the gain of the comprehensive health index relative to the historical periodic moving average. The exercise energy value is mapped to the virtual pet growth resource, and the virtual pet growth resource drives the accumulation of virtual pet experience, level increase, appearance unlocking and action slot expansion. The system acquires the user's real-time displacement vector, movement speed, and real-time movement intensity information to drive the virtual pet in the augmented reality scene to perform synchronous follow actions and status feedback interactions corresponding to the user's movement state. A digital twin of a user's health is constructed based on the historical change sequence of the comprehensive health index, and personalized exercise and dietary adjustment suggestions are output by combining the retrieval-enhanced generation model.

[0076] Example 5 If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A full-cycle motion tracking method based on multimodal data mining, characterized in that, Includes the following steps: Acquire multimodal health data of users within a preset assessment period; The multimodal health data is standardized to construct a health assessment index matrix, and the dynamic weight of each health assessment index in the current assessment period is calculated based on the health assessment index matrix. Based on dynamic weights, the health assessment indicator matrix is ​​weighted and the user's comprehensive health index for the current assessment period is calculated. The system acquires data from each segment of the user's current exercise process, calculates the baseline exercise value based on the intensity coefficient and effective exercise duration of each segment, and calculates the corresponding exercise energy value by combining the gain of the comprehensive health index relative to the historical periodic moving average. The exercise energy value is mapped to the virtual pet growth resource, and the virtual pet growth resource drives the accumulation of virtual pet experience, level increase, appearance unlocking and action slot expansion. The system acquires the user's real-time displacement vector, movement speed, and real-time movement intensity information to drive the virtual pet in the augmented reality scene to perform synchronous follow actions and status feedback interactions corresponding to the user's movement state. A digital twin of a user's health is constructed based on the historical change sequence of the comprehensive health index, and personalized exercise and dietary adjustment suggestions are output by combining the retrieval-enhanced generation model.

2. The full-cycle motion tracking method based on multimodal data mining according to claim 1, characterized in that, Multimodal health data includes exercise data, physiological data, and subjective feedback data. Exercise data includes exercise type, duration, intensity, trajectory, and estimated calorie expenditure. Physiological data includes height, weight, body fat percentage, resting heart rate, and sleep data. Subjective feedback data includes fatigue score and exercise mood score.

3. The full-cycle motion tracking method based on multimodal data mining according to claim 1, characterized in that, The specific method for calculating the dynamic weight of each health assessment indicator in the current assessment period based on the health assessment indicator matrix is ​​as follows: Obtain the health assessment indicator matrix, and perform missing value completion and outlier removal on each indicator in the health assessment indicator matrix; Based on preset standardization rules, dimensionless normalization is performed on each indicator; Calculate the corresponding information entropy value for each normalized indicator; Calculate the difference coefficient of the corresponding indicator based on the entropy value of each information; The dynamic weight of each health assessment indicator in the current assessment period is determined based on the proportion of each difference coefficient to the total of all difference coefficients.

4. The full-cycle motion tracking method based on multimodal data mining according to claim 1, characterized in that, The specific method for calculating a user's overall health index during the current assessment period is as follows: Obtain the weighted health assessment index matrix, and then weight each index in the health assessment index matrix according to dynamic weights to generate a weighted evaluation matrix. The maximum value of each indicator in the weighted evaluation matrix is ​​extracted as the positive ideal solution, and the minimum value is extracted as the negative ideal solution. Calculate the first distance between the user's current evaluation data and the ideal solution; Calculate the second distance between the user's current evaluation data and the negative ideal solution; The comprehensive health index is determined based on the ratio of the second distance to the sum of the first distance and the second distance.

5. The full-cycle motion tracking method based on multimodal data mining according to claim 1, characterized in that, The specific method for calculating the corresponding exercise energy value, based on the gain of change in the comprehensive health index relative to the historical periodic moving average, is as follows: Obtain the change gain of the historical periodic moving average, and divide the user's current motion process into motion segments according to the continuous motion duration change and motion state switching nodes; Based on the exercise type and real-time heart rate zone corresponding to each exercise segment, the intensity coefficient is matched accordingly. The intensity coefficient of each exercise segment is multiplied by the corresponding effective exercise duration and then summed to obtain the baseline exercise value. The gain on health change is calculated based on the comprehensive health index of the current assessment period and the moving average within a preset historical period window. The exercise energy value corresponding to the current exercise is generated based on the product of the baseline exercise value and the health change gain.

6. The full-cycle motion tracking method based on multimodal data mining according to claim 1, characterized in that, The specific method for obtaining the user's real-time displacement vector, movement speed, and real-time movement intensity information to drive the virtual pet in the augmented reality scene to perform synchronized following actions and status feedback interactions corresponding to the user's movement state is as follows: Acquire real-time displacement vector, motion velocity, and real-time motion intensity information of the user, and identify target planes in the real environment based on images from the mobile terminal's camera. Establish augmented reality anchor point coordinates on the target plane; The virtual pet's position relative to the augmented reality anchor point is updated based on the user's real-time displacement vector. Match the virtual pet's walking, running, or jumping animations to the user's movement speed; The virtual pet's facial expressions and voice feedback are switched according to the user's real-time exercise intensity.

7. The full-cycle motion tracking method based on multimodal data mining according to claim 1, characterized in that, When a user submits a transaction request, the item identifier, validity period status, and listing energy value of the virtual item to be traded are written into the transaction queue. Transaction matching is performed based on item type, listed energy value, and user purchase conditions; After a successful match, the corresponding exercise energy value will be deducted from the buyer's account and written to the seller's account; The ownership identifiers of traded items are updated synchronously.

8. A full-cycle motion tracking system based on multimodal data mining, characterized in that, include: The data acquisition module is used to acquire the user's multimodal health data within a preset assessment period; The dynamic weighting module is used to standardize multimodal health data, construct a health assessment indicator matrix, and calculate the dynamic weight of each health assessment indicator in the current assessment period based on the health assessment indicator matrix. The comprehensive index module is used to perform weighted processing on the health assessment indicator matrix based on dynamic weights and calculate the user's comprehensive health index within the current assessment period. The exercise energy module is used to acquire data of each exercise segment during the user's current exercise process, calculate the basic exercise value based on the intensity coefficient and effective exercise duration of each exercise segment, and calculate the corresponding exercise energy value by combining the change gain of the comprehensive health index relative to the historical periodic moving average. The pet enhancement module maps exercise energy values ​​to virtual pet growth resources, which drive the accumulation of virtual pet experience, level increase, appearance unlocking, and action slot expansion based on virtual pet growth resources; The interaction module is used to obtain the user's real-time displacement vector, movement speed and real-time movement intensity information, and drive the virtual pet in the augmented reality scene to perform synchronous following actions and status feedback interactions corresponding to the user's movement state; The output module is used to construct a digital twin of the user's health based on the historical change sequence of the comprehensive health index, and output personalized exercise suggestions and dietary adjustment suggestions in combination with the retrieval enhancement generation model.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the full-cycle motion tracking method based on multimodal data mining as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the full-cycle motion tracking method based on multimodal data mining as described in any one of claims 1 to 7.

Images