Health management data analysis method and system combined with reinforcement learning and intelligent chip

Abstract

The application discloses a health management data analysis method and system combined with reinforcement learning and an intelligent chip, and relates to the technical field of health management data analysis. The application discloses a health management data analysis method and system combined with reinforcement learning and an intelligent chip, and relates to the technical field of health management data analysis. The application solves the technical problem that the reinforcement learning immediate reward mechanism cannot accurately attribute the contribution of historical actions due to the non-uniform dimensions of multi-source data and the time delay of intervention effects in the existing health management data analysis.

Description

Technical Field

[0001] This invention relates to the field of health management data analysis technology, specifically to a health management data analysis method, system, and intelligent chip that incorporates reinforcement learning. Background Technology

[0002] With the widespread adoption of wearable devices, mobile health applications, and smart medical devices, users' health intervention execution data and physiological indicator data have been collected with unprecedented precision. This data provides rich raw material for personalized health management decisions, and how to accurately recommend the optimal combination of health intervention programs to different users based on this data has become a research hotspot in the field of health management data analysis.

[0003] However, existing methods struggle to effectively address issues related to measurement standards, temporal correspondence, and delay effects in multi-source intervention data. They also lack the ability to adapt to baseline differences among individuals, resulting in insufficient accuracy in assessing intervention effects and hindering precise, personalized health management decisions. Summary of the Invention

[0004] This invention provides a health management data analysis method, system, and intelligent chip that incorporates reinforcement learning. This invention solves the technical problem in existing health management data analysis where inconsistent dimensions of multi-source data and time delays in intervention effects prevent reinforcement learning's immediate reward mechanism from accurately attributing contributions to historical actions.

[0005] In view of the above problems, the present invention provides a health management data analysis method combining reinforcement learning, the method comprising: Obtain users' historical execution records and time-series data of health indicators across various health intervention programs; Calculate the periodic intervention execution volume of each health intervention program and the periodic health indicator change of each health indicator in each monitoring period; By conducting cross-period time lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator, the time lag vector of the program effect of each health intervention program on each health indicator is determined. Based on the time lag vector of the scheme's effect, the cross-cycle effect attribution is performed on the changes in the periodic health indicators of each monitoring period to obtain the cumulative effect contribution of multiple schemes. In the reward allocation of the reinforcement learning model, the reward for changes in health indicators in the current period is distributed in reverse along the time axis to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme's effect, and the decision-making strategy is updated.

[0006] This invention also provides a health management data analysis system incorporating reinforcement learning, comprising: The data acquisition module is used to acquire users' historical execution records and time-series data of health indicators for various health intervention programs; The periodic quantification module is used to calculate the periodic intervention execution volume of each health intervention program and the periodic health indicator change volume of each health indicator in each monitoring period. The time lag analysis module is used to determine the time lag vector of the effect of each health intervention program on each health indicator by performing cross-period time lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator. The cross-cycle attribution module is used to perform cross-cycle effect attribution on the changes in the periodic health indicators of each monitoring period based on the time lag vector of the scheme effect, and to obtain the cumulative effect contribution of multiple schemes. The reinforcement learning strategy update module is used to revert the reward for changes in health indicators in the current period to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme effect, in the reward allocation of the reinforcement learning model, and update the decision-making strategy.

[0007] The present invention also provides a smart chip, comprising: Memory, used to store computer programs; A processor is used to read and execute the computer program, thereby enabling a health management data analysis method that incorporates reinforcement learning.

[0008] One or more technical solutions provided in this invention have at least the following technical effects or advantages: This invention eliminates temporal misalignment by aligning execution records and health indicator data from different sources along the timeline and filling in missing data. It eliminates dimensional barriers between multiple intervention programs by dividing the actual number of executions by the required number. Furthermore, it removes individual static baselines by subtracting the current period's detection value from the previous period's detection value, significantly reducing the probability of spurious correlations caused by dimensional inconsistencies and baseline drift in subsequent analyses. By performing a full-range time-shift traversal on the paired combinations of each intervention program and health indicator and independently calculating the Pearson correlation coefficient at each time shift, the constructed program effect lag vector fully records the distribution of all delayed correlation strengths, fundamentally solving the problem of causal misjudgment caused by neglecting delayed effects in traditional contemporaneous analysis methods. By normalizing the rewards for changes in health indicators in the current period and distributing them to each historical monitoring period according to the attribution coefficient, the complete transmission and conservation of reward signals over time are ensured. This enables the reinforcement learning model to accurately identify historical intervention actions that actually produce effects, solving the attribution error caused by the neglect of delayed effects in traditional instant reward mechanisms. At the same time, by dynamically calculating the adaptive learning rate based on the cumulative effect contribution and constructing a reward function that takes into account both short-term effects and long-term contributions, the model's convergence speed and training stability are effectively improved. The final output decision strategy can achieve precise adaptive closed-loop health management based on individual user historical feedback.

[0009] In summary, this invention solves the technical problem in existing health management data analysis where inconsistent dimensions of multi-source data and time delays in intervention effects prevent reinforcement learning's immediate reward mechanism from accurately attributing contributions to historical actions. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the health management data analysis method combining reinforcement learning provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the health management data analysis system combining reinforcement learning provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the smart chip provided in an embodiment of the present invention; The components represented by each number in the attached diagram are explained below: Data acquisition module 11, periodic quantization module 12, time delay analysis module 13, cross-period attribution module 14, reinforcement learning strategy update module 15, intelligent chip 200, memory 210, computer program 211, processor 220. Detailed Implementation

[0012] This invention provides a health management data analysis method, system, and intelligent chip that combines reinforcement learning. It specifically addresses the technical problem in existing health management data analysis where the inconsistent dimensions of multi-source data and the time delay in intervention effects prevent reinforcement learning's immediate reward mechanism from accurately attributing contributions to historical actions.

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0014] Example 1, as Figure 1 As shown, this invention provides a health management data analysis method combining reinforcement learning, the method comprising: S100: Obtain the user's historical execution records and time-series data of health indicators for various health intervention programs.

[0015] In existing health management data collection technologies, data from different intervention programs such as exercise, diet, and medication originate from independent systems including wearable devices, user-generated records, and smart pillboxes. The recording formats of these data sources are inconsistent, and the collection time points are scattered and misaligned, making it impossible to directly establish an accurate temporal correspondence between the execution status of multiple intervention types within the same period and changes in health indicators. Furthermore, existing collection methods only record the actual number of executions, losing crucial baseline information such as the required number of executions as prescribed by the doctor. This results in a lack of a fundamental reference for judging whether user compliance is up to standard. Since reinforcement learning models require input data to be precisely aligned on the timeline and to fully contain the correspondence between states and actions for effective policy learning and reward reversal, how to construct a data acquisition method that can comprehensively collect execution information of multiple intervention programs and multi-dimensional health indicator time-series data while ensuring precise temporal alignment of various data sources has become a pressing technical problem to be solved.

[0016] Step S100 in the method provided in this embodiment of the invention includes: Obtain the health intervention program execution records of the user in each monitoring cycle within a preset historical time period. Each execution record includes the intervention program type, the number of times it should be executed, and the number of times it is actually executed. The intervention program type includes at least two of the following: exercise intervention, dietary intervention, and medication intervention. The health indicator data of the user in each monitoring period within the preset historical time period are obtained, wherein the health indicator includes at least one of weight, blood sugar and blood pressure, and the health indicator data are arranged in chronological order according to the monitoring period to form health indicator time series data.

[0017] In this embodiment of the invention, the system first connects to the user's health management database, the backend server of a mobile health application, or a cloud data warehouse via wired or wireless communication to extract the user's health intervention program execution records within a preset historical period. The preset historical period is set according to the target chronic disease management cycle, with a typical value of the past ninety days. The timeline is discretized into fixed monitoring cycles, with each monitoring cycle set to one day, meaning every twenty-four hours constitutes a monitoring cycle.

[0018] Each execution record includes the intervention type, the number of times it should have been executed, and the actual number of times it was executed. The intervention type must include at least two of the following: exercise intervention, dietary intervention, and medication intervention. Exercise intervention refers to a program that uses physical activity to consume energy and improve metabolic function; it can be further specified as a daily walking plan or resistance training plan. Dietary intervention refers to a program that influences physiological indicators by regulating nutrient intake and total calorie intake; it can be further specified as a daily calorie intake control plan or a nutritional meal plan. Medication intervention refers to a program that directly regulates physiological functions through drug intake; it can be further specified as an oral hypoglycemic drug administration plan or an antihypertensive drug administration plan.

[0019] The required number of executions is determined by the doctor's prescription or the quantitative goals in the system's preset management plan. For example, if the doctor requires three walks per day or two tablets of hypoglycemic medication per day, this field records the corresponding value. The actual number of executions is obtained through comprehensive statistics from multi-source sensing devices and user interaction records. For exercise interventions, the actual number of executions is calculated by using accelerometer and gyroscope data from the user's smart bracelet or sports watch to determine the effective exercise period. For dietary interventions, the actual number of executions is obtained by summarizing the user's pre-meal photo check-in combined with image recognition technology or manually entered meal records. For medication interventions, the actual number of executions is obtained by recording each medication dispensing action using the contact sensor built into the smart pillbox or by having the user scan the drug's QR code to confirm medication administration. The various types of intervention plans within each monitoring cycle are recorded to form an original execution record matrix.

[0020] Furthermore, within the same preset historical time period, user health indicator data are acquired synchronously at the same monitoring cycle scale. Health indicators are core physiological parameters for specific chronic disease management, including at least one of weight, fasting blood glucose, and blood pressure. Weight data is automatically measured and uploaded via Bluetooth using a smart body fat scale in the user's fasting state in the morning, measured in kilograms. Fasting blood glucose data is automatically measured and recorded using a home blood glucose meter before the user eats in the morning, measured in millimoles per liter. Blood pressure data is measured using an electronic blood pressure monitor after the user has rested for fifteen minutes, including both systolic and diastolic blood pressure components, measured in millimeters of mercury. For each health indicator, the measured values ​​within each monitoring cycle are arranged chronologically from the initial monitoring cycle to the current monitoring cycle, constructing a one-dimensional time-series data structure. For example, if the preset historical period is ninety days, the blood glucose time series data contains ninety data points from the first day to the ninetieth day. Each data point is mapped one-to-one with the serial number of its monitoring period, and is represented in array form. The array index represents the position relative to the monitoring period.

[0021] Finally, the acquired historical execution records and health indicator time-series data are timestamped and verified to ensure that each execution record corresponds one-to-one with the health indicator data of its corresponding monitoring period in the time dimension. If data is missing in a certain monitoring period, linear interpolation or forward imputation is used to fill in the missing data, and the data for that period is marked as imputed data to ensure the integrity of the time series and the effectiveness of subsequent cross-period analysis.

[0022] This invention, by limiting historical execution records to include intervention type, required execution count, and actual execution count, ensures that the raw data acquired by the system fully retains information from both the medical order baseline and actual execution dimensions. This provides an indispensable reference for subsequent compliance quantification and avoids the problem of losing evaluation criteria due to collecting only absolute values. By limiting health indicators to at least one of weight, fasting blood glucose, and blood pressure, the acquired data covers multiple core physiological levels such as metabolism, endocrine function, and circulation, providing multi-dimensional state perception signals for reinforcement learning models. More importantly, by using the monitoring cycle number as an anchor point to perform time alignment and missing data completion on execution record data and health indicator data from different sources, the temporal misalignment caused by differences in acquisition equipment and inconsistent reporting times is eliminated, ultimately generating a structured dataset with complete cycles and consistent temporal sequence.

[0023] S200: Calculate the periodic intervention execution amount of each health intervention program and the periodic health indicator change amount of each health indicator in each monitoring period.

[0024] Different intervention programs, such as exercise, diet, and medication, are measured in different physical units, such as frequency, meal frequency, and number of tablets. These absolute values ​​cannot be compared or jointly analyzed within the same quantitative framework. Furthermore, the medical needs of different users vary greatly, and the absolute value of the actual number of times the medication was administered is insufficient to determine whether the user has met the required adherence standards. This results in the loss of crucial information regarding the reference benchmark for doctor's prescriptions in the raw data. In addition, the raw absolute values ​​of health indicators, such as fasting blood glucose and weight, are greatly affected by individual differences in users' innate constitution and basal metabolic rate. Directly using time-series absolute values ​​for cross-period correlation analysis introduces significant individual baseline shift noise, leading to spurious correlations from numerous non-intervention factors in the analysis results, failing to accurately reflect the net causal effect between intervention implementation and health changes. Therefore, how to convert the collected raw absolute values ​​into a standardized dynamic feature sequence that eliminates dimensional differences and baseline drift while possessing clear directional meaning has become a pressing technical problem to be solved.

[0025] Step S200 in the method provided in this embodiment of the invention includes: Obtain the number of times each health intervention plan should be executed and the number of times it is actually executed in each monitoring period. Divide the number of times it is actually executed by the number of times it should be executed to obtain the periodic intervention execution amount of the health intervention plan in the monitoring period. Based on the time-series data of the health indicators, the detection value of each health indicator in the current monitoring period and the detection value in the previous monitoring period are obtained. The detection value in the previous monitoring period is subtracted from the detection value in the current monitoring period to obtain the periodic health indicator change in the current monitoring period.

[0026] In this embodiment of the invention, for each monitoring cycle and each health intervention plan within a preset historical time period, the required execution count and the actual execution count for that plan within that cycle are read respectively. The required execution count comes from the medical orders recorded in the doctor's prescription or the quantitative target value specified in the system's preset management plan. The actual execution count comes from the actual completion value obtained through comprehensive statistical analysis of wearable devices, dietary check-in records, or smart pillbox sensor data. For each health intervention plan within each monitoring cycle, the actual execution count of that plan within that cycle is divided by the required execution count to obtain the cycle intervention execution volume of that health intervention plan in that monitoring cycle. The cycle intervention execution volume is a dimensionless real number, and its value range is usually between zero and one. When the value is equal to one, it means that the user has fully followed the medical orders; when the value is less than one, it means that the execution volume is insufficient; and when the value is greater than one, it means that the execution volume is excessive. The above division operation is performed independently for each intervention program type and each monitoring period to generate a periodic intervention execution volume matrix with the monitoring period number as the row index and the intervention program type as the column index. Each element in the matrix represents the relative degree of completion of a specific intervention program within a specific monitoring period.

[0027] Furthermore, based on the acquired one-dimensional time series data of each health indicator, a difference operation is performed on each health indicator. For each health indicator, the detection value of the current monitoring period is first extracted from the time series data of that indicator, and simultaneously, the detection value of the previous adjacent monitoring period is extracted. If the current monitoring period is the first monitoring period in the entire preset historical period, since there is no previous monitoring period, the change in the periodic health indicator in the current monitoring period is marked as invalid and excluded from subsequent analysis. For all monitoring periods other than the first monitoring period, the system performs a subtraction operation, using the detection value of the previous monitoring period as the minuend and the detection value of the current monitoring period as the subtrahend. The difference between the detection value of the previous monitoring period and the detection value of the current monitoring period is the change in the periodic health indicator in the current monitoring period.

[0028] Specifically, a positive difference indicates that the health indicator has improved compared to the previous period. For example, if fasting blood glucose decreases from 8 mmol / L in the previous period to 7 mmol / L in the current period, the difference is +1 mmol / L, representing an improvement in blood glucose levels. A negative difference indicates that the health indicator has deteriorated compared to the previous period. For example, if weight increases from 70 kg in the previous period to 71 kg in the current period, the difference is -1 kg, representing a deterioration in weight control. The above subtraction operation is performed independently on the time-series data of each health indicator for each period, generating a matrix of periodic health indicator changes indexed by monitoring period number (row index) and health indicator type (column index). Each element in this matrix represents the short-term dynamic drift direction and magnitude of a specific health indicator within a specific monitoring period. The aforementioned periodic intervention execution matrix and periodic health indicator change matrix together constitute the standard numerical input for subsequent cross-period time-lag correlation analysis.

[0029] This invention eliminates the dimensional barriers caused by different physical units of measurement among exercise, diet, and medication by dividing the actual number of executions by the required number of executions to obtain the periodic intervention execution quantity. This provides a unified measurement scale for the execution intensity of various intervention programs, and the ratio directly anchors the benchmark requirements of doctors' prescriptions, giving user compliance clinical reference significance. By subtracting the current period's detection value from the previous period's detection value, the change in periodic health indicators is obtained, successfully separating each user's individual static baseline level. The positive or negative sign of the change directly and clearly indicates the direction of improvement or deterioration in health status. After the above two transformations, a high-quality input sequence with zero mean, stationarity, and clear causal direction is provided for cross-period time-lag correlation analysis. This significantly reduces the probability of spurious correlations caused by dimensional inconsistencies and baseline drift, thereby effectively improving the statistical authenticity and clinical interpretability of the program effect time-lag vector.

[0030] S300: By performing cross-period time lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator, the time lag vector of the program effect of each health intervention program on each health indicator is determined.

[0031] Current health management data analysis techniques generally calculate the direct correlation between the amount of intervention implemented and the changes in health indicators during the same period, assuming by default that the intervention effect is immediately apparent within the implementation period. However, the effects of exercise intervention on weight can only be observed after several days, the effects of dietary intervention on blood glucose have a delay of several hours to several days, and drug intervention requires pharmacokinetic processes of absorption, distribution, metabolism, and excretion before it can produce a measurable effect on the target indicators. This time delay makes contemporaneous correlation analysis fundamentally unable to truly reflect the actual causal relationship between intervention and effect.

[0032] Meanwhile, different intervention programs have varying delay periods for different health indicators, and the delay characteristics of the same program also differ significantly for different indicators. Existing fixed lag analysis methods lack a systematic approach to describe the distribution of the correlation strength between each program and each indicator across the entire delay range. Therefore, how to construct a quantitative expression vector that can fully describe the transition from immediate to delayed effects for each pairing combination of health intervention programs and each health indicator has become a pressing technical problem that needs to be solved.

[0033] Step S300 in the method provided in this embodiment of the invention includes: For each combination of the health intervention program and each health indicator, obtain the periodic intervention execution sequence of the program and the periodic health indicator change sequence of the health indicator, wherein the periodic intervention execution sequence and the periodic health indicator change sequence contain the same number of monitoring periods. The time shift range is set to multiple time shift values ​​between zero and a preset maximum time shift, wherein the preset maximum time shift is the longest delay period in which the health intervention plan may produce an observable effect on health indicators; For each time-shifted value, the sequence of changes in the periodic health indicators is shifted forward by the number of periods corresponding to the time-shifted value, so that the shifted sequence of changes in indicators is aligned period by period with the portion of the unshifted sequence of periodic intervention execution that does not exceed the shift range. Calculate the Pearson correlation coefficient of the aligned portion sequence, and use it as the time-delay correlation coefficient corresponding to the time-shift value; Arrange the time-lag correlation coefficients corresponding to all time-shift values ​​in ascending order of time-shift value to form the time-lag vector of the health intervention program's effect on the health indicator.

[0034] In this embodiment of the invention, after calculating the periodic intervention execution volume and the periodic health indicator changes for each monitoring period, a cross-period time-lag correlation analysis is performed on each combination of health intervention program and each health indicator. The goal of this analysis is to quantify the delayed correlation strength between the execution volume of a certain program in a certain historical period and the changes in health indicators in the current period.

[0035] Specifically, for each combination of health intervention program and health indicator, the cyclical intervention execution volume sequence of the program across all monitoring periods is first obtained, along with the cyclical health indicator change sequence of the health indicator across all monitoring periods. These two sequences contain the same number of monitoring periods, and the monitoring period numbers correspond one-to-one; that is, data at the same index position in both sequences originate from the same monitoring period.

[0036] Furthermore, the time shift is set to any integer value from zero to the preset maximum time shift. The preset maximum time shift is a predetermined positive integer representing the longest delay period during which the health intervention program may produce observable effects on health indicators from the time of implementation. The value of the preset maximum time shift is set according to the mechanism of action of different intervention programs. For example, the effects of exercise and dietary interventions on blood glucose and weight usually appear within several days, so their preset maximum time shift can be set to seven days. Due to the pharmacokinetic characteristics of drug interventions, their preset maximum time shift can be determined based on the time to peak effect of the drug, the lag characteristics of the effect, or the time of the maximum effect observed in clinical studies, with a typical range of three to fourteen days. In addition, the value of the preset maximum time shift must not exceed one less than the total number of periods in the cyclical intervention execution sequence to ensure that there are enough data points for alignment in subsequent calculations.

[0037] Furthermore, for each time shift value within the preset maximum time shift range, a sequence shift and alignment operation is performed. Specifically, the sequence of periodic health indicator changes is shifted forward along the time axis by the number of periods corresponding to that time shift value. Forward shifting means reassigning each indicator change value in the sequence that was originally located in a certain monitoring period to the position of the next period after that time shift value, so that a new correspondence is formed between the shifted indicator change sequence and the unshifted periodic intervention execution sequence. Under this new correspondence, the intervention execution value located in the t-th monitoring period will be paired with the health indicator change value located in the t+d-th monitoring period, where d is the time shift value, representing the number of periods after which the health indicator response is observed. During alignment, only data points that do not exceed the translation range participate in subsequent calculations. That is, in the untranslated cyclic intervention execution sequence, the part from the first monitoring period to the N minus d monitoring period is aligned one by one with the part from the d plus one monitoring period to the Nth monitoring period in the translated indicator change sequence, where N is the total number of monitoring periods. Data points that exceed the translation range are discarded due to the lack of matching objects.

[0038] Furthermore, for each pair of aligned subsequences, the Pearson correlation coefficient is calculated between them. The Pearson correlation coefficient is calculated by dividing the covariance of the two sequences by the product of their respective standard deviations. The result is a real number between negative and positive one. A positive value indicates a positive correlation, meaning that the greater the intervention, the greater the improvement in the corresponding indicator. A negative value indicates a negative correlation. This calculated result is the time-lag correlation coefficient corresponding to that time shift value.

[0039] Finally, for each integer time shift within the preset maximum time shift range, the above translation, alignment, and correlation coefficient calculation operations are performed sequentially. After completion, a set of time-lag correlation coefficients is obtained, the number of which equals the value of the preset maximum time shift plus one. This set of time-lag correlation coefficients is arranged in ascending order of time shift value; that is, the first element corresponds to the correlation coefficient when the time shift value is zero, the second element corresponds to the correlation coefficient when the time shift value is one, and so on, until the time shift value corresponds to the correlation coefficient of the preset maximum time shift. The numerical vector formed by this arrangement is the time-lag vector of the health intervention plan's effect on the health indicator. Each element of this vector represents the correlation strength under a specific delay period, and the vector as a whole reflects the complete delayed correlation distribution characteristics between intervention execution and indicator response.

[0040] This invention, through a full traversal of time shifts from zero to a preset maximum for each paired combination of health intervention and health indicator, and independently calculating the Pearson correlation coefficient between the shifted sequences for each time shift value, constructs a time lag vector that comprehensively records the entire distribution of correlation strength from immediate response to maximum delayed response. This fully reveals the differences in the speed of onset, peak effect time, and duration of effect of different interventions on different health indicators, providing precise time lag quantification weights for subsequent cross-period effect attribution. This time lag vector allows the attribution process to obtain differentiated weighting coefficients based on the actual delay between each historical period and the current period, fundamentally solving the problem of causal misjudgment caused by neglecting delayed effects in traditional contemporaneous analysis methods, and significantly improving the clinical authenticity and physiological rationality of attribution analysis.

[0041] S400: Based on the time lag vector of the scheme effect, perform cross-cycle effect attribution on the changes in the periodic health indicators of each monitoring period to obtain the cumulative effect contribution of multiple schemes.

[0042] Current technologies for analyzing health management data typically only correlate interventions performed during the same period with changes in health indicators during the same period, completely ignoring the core characteristic that intervention effects have varying time delays. Even though some technologies recognize the existence of the delay effect and attempt to analyze the time lag relationship between a single program and a single indicator through methods such as translation and alignment, there is a lack of an attribution quantification method that can comprehensively integrate the execution volume of multiple intervention programs across multiple historical monitoring periods within a preset retrospective window with their respective differentiated time lag weights. This results in an inability to accurately assess the overall net contribution of all historical intervention behaviors to the current changes in health indicators in terms of total data.

[0043] Meanwhile, without clear constraints on the rules governing the values ​​of elements in the time-delay vector, the system will be unable to accurately extract the corresponding weight values ​​based on the calculated number of time-delay cycles during the attribution process. Therefore, how to construct a multi-scheme cumulative attribution calculation method across cycles, so that the comprehensive contribution of all historical interventions can be quantified into a scalar value for subsequent reinforcement learning reward allocation, has become a pressing technical problem that needs to be solved.

[0044] Step S400 in the method provided in this embodiment of the invention includes: Obtain the periodic intervention execution volume of each health intervention plan in each historical monitoring period within a preset traceability window prior to the current monitoring period, wherein the length of the preset traceability window is not less than the preset maximum time shift; For each combination of the health intervention program and each health indicator, obtain the time lag vector of the program effect of the health intervention program on the health indicator; For each historical monitoring cycle within the preset traceability window, the number of cycles of intervention execution of the health intervention plan in the historical monitoring cycle is obtained, and the difference between the current monitoring cycle and the historical monitoring cycle is calculated to obtain the corresponding time lag cycle number. The time-shift value of the time-delay vector of the scheme effect is extracted as the time-delay correlation coefficient corresponding to the time-delay period number, and used as the attribution coefficient corresponding to the historical monitoring period; Multiply the periodic intervention execution amount by the attribution coefficient, and the resulting product is used as the attribution contribution component of the health intervention program to the health indicator within the historical monitoring period; The attribution contribution components of all historical monitoring periods and all health intervention programs within the preset traceability window are added together, and the sum is used as the cumulative effect contribution of the health indicator across multiple programs.

[0045] The effect time delay vector of the proposed scheme is a multi-dimensional vector, and the k-th element of the vector is the time delay correlation coefficient when the time shift value is k periods.

[0046] In this embodiment of the invention, after constructing the time lag vector of the effect of each health intervention program on each health indicator, the time lag vector is used to perform cross-cycle effect attribution on the change of the periodic health indicator in each monitoring period, so as to quantify the extent to which the currently observed change of health indicator is contributed by the execution of various intervention programs in previous historical monitoring periods.

[0047] Specifically, a preset tracing window is first determined. The length of this window is measured in units of monitoring periods, representing the total number of historical periods traced back from the current monitoring period. The length of the preset tracing window is pre-set, and its value must not be less than a preset maximum time shift to ensure that all historical interventions that may affect the current health indicator change are within the tracing coverage. For example, if the preset maximum time shift is seven periods, then the length of the preset tracing window must be at least seven periods to ensure that all historical interventions delayed from one period to seven periods are included in the attribution calculation.

[0048] Furthermore, for each historical monitoring period within a pre-defined retrospective window prior to the current monitoring period, attribution component calculations are performed for each combination of each health intervention program and each health indicator. Specifically, firstly, the periodic intervention execution volume of the health intervention program in that historical monitoring period is obtained. Then, the difference between the sequence number of the current monitoring period and the sequence number of the historical monitoring period is calculated. This difference represents the time interval between the historical monitoring period and the current monitoring period, and the number of periods corresponding to this interval is the time lag period number corresponding to the historical monitoring period.

[0049] Subsequently, the time-lag vector of the health intervention program's effect on the health indicator is obtained. It should be noted that the time-lag vector is a multi-dimensional vector, and the k-th element of this vector is the time-lag correlation coefficient with a time shift of k periods. Based on the calculated number of time lag periods, the element value corresponding to that number of time lag periods is extracted from the time-lag vector. Specifically, if the number of time lag periods is d, the d-th element is extracted from the time-lag vector, and this element is the time-lag correlation coefficient corresponding to that number of time lag periods. The extracted time-lag correlation coefficient is used as an attribution coefficient in the historical monitoring period, representing the contribution weight of the intervention implementation amount in that historical period to the current change in the health indicator.

[0050] Furthermore, multiplying the obtained periodic intervention execution volume by the extracted attribution coefficient yields the attribution contribution component of the health intervention program to the health indicator within the historical monitoring period. When the periodic intervention execution volume is large and the attribution coefficient is a large positive value, the attribution contribution component is also large, indicating that the sufficient execution of the program in the historical period has a significant positive contribution to the current health improvement. When the attribution coefficient is negative, the attribution contribution component is negative, indicating that the execution of the program in the historical period shows a correlation contrary to health improvement within the current observation window, that is, the more it is executed, the worse the indicator becomes. When the periodic intervention execution volume is close to zero, the attribution contribution component is also close to zero, indicating that the intervention execution in the historical period is insufficient, and even if the attribution coefficient is large, it cannot produce a substantial contribution.

[0051] Finally, for all historical monitoring periods, all types of health intervention programs, and all types of health indicators within the preset traceability window, the aforementioned attribution contribution components were calculated independently. After all calculations were completed, the attribution contribution components corresponding to all historical monitoring periods, all health intervention programs, and all health indicators were summed. This summation value represents the cumulative effect contribution of the health indicator across multiple periods for the current monitoring period. This cumulative effect contribution is a comprehensive scalar, and its magnitude and positive / negative direction comprehensively reflect the overall contribution of all historical intervention activities within the preset traceability window to the change in the current health indicator.

[0052] If multiple health intervention programs have a positive attributable contribution to the same health indicator, the cumulative effect contribution is the algebraic sum of the positive contributions of each program. If some programs have negative attributable contributions, these negative contributions will partially offset the positive contributions during the accumulation process, and the final cumulative effect contribution reflects the net effect after the combined effect of all programs. The above calculation process for the cumulative effect contribution is performed independently for each health indicator.

[0053] This invention, through independent calculation of the attribution contribution component for each intervention scheme in each historical monitoring period within a preset tracing window and summing all contribution components, successfully quantifies the combined impact of multiple historical moments and various types of interventions into a single scalar value. This allows the contributions of different schemes to be algebraically superimposed within a unified attribution framework, reflecting a net effect. By clearly defining the rules for the values ​​of the elements in the scheme effect time-lag vector, the system accurately extracts the corresponding time-lag correlation coefficient from the vector based on the number of time-lag periods as attribution weights. This ensures that the intervention execution volume in each historical period receives differentiated weighting according to its true delay characteristics, thus providing a clear time-lag causal basis for calculating the cumulative effect contribution. The final output scalar directly serves the construction of the reward function in reinforcement learning, providing a quantitative attribution basis for subsequent reverse distribution of rewards along the time axis.

[0054] S500: In the reward allocation of the reinforcement learning model, the reward for the change in health indicators in the current period is distributed in reverse along the time axis to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme effect, and the decision-making strategy is updated.

[0055] In existing reinforcement learning-based health management decision-making techniques, reward signals are typically set only based on immediate changes in health indicators within the current period. Positive rewards are given for improved health, and negative rewards for deterioration. This reward mechanism, which relies entirely on immediate feedback, ignores the inherent delay in the effects of health interventions. This leads the model to incorrectly attribute current period health improvements to the current intervention during policy updates, while historical interventions that actually produced effects fail to receive the necessary positive reinforcement, causing a deviation in the direction of policy updates. Furthermore, existing reinforcement learning models use a fixed learning rate during training, failing to dynamically adjust the step size of policy updates based on the long-term net performance of historical interventions, thus reducing the model's convergence efficiency at different policy performance stages.

[0056] Furthermore, existing technologies lack a method to systematically transform cross-period attribution analysis results into training sample labels and construct corresponding reward functions, creating a disconnect between attribution analysis conclusions and the reinforcement learning training process, hindering effective integration. Therefore, how to construct a method that can retroactively allocate rewards for current health changes to historical intervention actions along the timeline and adaptively update the strategy accordingly, and how to integrate attribution results into the reinforcement learning model training framework, have become pressing technical problems that need to be solved.

[0057] Step S500 in the method provided in this embodiment of the invention includes: Obtain the change in the health indicator during the current monitoring period, and use the change in the health indicator as a reward for the change in the health indicator during the current period. For each historical monitoring period within a preset traceability window prior to the current monitoring period, the intervention actions performed in the historical monitoring period are obtained, and the attribution weight of each health intervention program in the intervention actions of the historical monitoring period on the reward for the change of health indicators in the current monitoring period is determined according to the attribution coefficient. Multiply the reward for changes in the health indicators by the attribution weight to obtain the cross-period allocation reward value of the health intervention program within the historical monitoring period; The cross-cycle allocation reward value for each historical monitoring period within the preset traceability window is added to the instant reward value for each historical monitoring period to obtain the corrected reward value for each historical monitoring period. The adaptive learning rate for the current update round is calculated based on the standard learning rate of the reinforcement learning model and the cumulative effect contribution of the multiple schemes. With the goal of maximizing the cumulative corrected reward value, the decision strategy of the reinforcement learning model is updated using the adaptive learning rate.

[0058] In this embodiment of the invention, after calculating the cumulative effect contribution of multiple schemes across cycles, the contribution is integrated into the reward allocation mechanism of the reinforcement learning model, so that the changes in health indicators observed in the current monitoring cycle can be propagated backward along the time axis to the intervention actions of the historical cycles that produced the effect, thereby guiding the updating of decision-making strategies.

[0059] Specifically, the change in health indicators for the current monitoring period is obtained. This change directly reflects the direction and magnitude of improvement or deterioration of health indicators compared to the previous period. The change in health indicators for this period is used as a reward for health indicator changes in the current period. This reward is a signed numerical value, with positive values ​​representing positive incentives for health improvement and negative values ​​representing negative penalties for health deterioration.

[0060] Furthermore, for each historical monitoring period within the preset traceability window prior to the current monitoring period, a cross-period reward allocation operation is performed. For each historical monitoring period, the various health intervention actions performed within that period are acquired, and the attribution weight of each health intervention plan within that historical monitoring period on the reward for changes in health indicators in the current monitoring period is determined. When determining the attribution weight, the attribution coefficient corresponding to each health intervention plan within each historical monitoring period is first obtained according to the aforementioned method. This attribution coefficient is derived from the time lag correlation coefficient extracted from the time lag vector of the plan's effect based on the number of time lag periods between that period and the current period.

[0061] Then, the absolute values ​​of the attribution coefficients for all historical monitoring periods and all health intervention programs within the preset traceability window are calculated, and these absolute values ​​are summed to obtain a total absolute value denominator. For each specific attribution coefficient, its absolute value is divided by this total absolute value denominator, and the resulting ratio is the attribution weight corresponding to that health intervention program within that historical monitoring period. This weight is a non-negative real number normalized to the interval between zero and one, and the sum of the attribution weights for all historical monitoring periods and all health intervention programs within the preset traceability window equals one.

[0062] The weight reflects the proportion of the absolute value of the attribution coefficient of a specific intervention in a particular historical period to the sum of the absolute values ​​of the attribution coefficients of all historical interventions. A larger proportion indicates a greater contribution of the intervention to the current health indicator change in that period. When the attribution coefficient is positive, its corresponding attribution weight is used to allocate a positive reward share, indicating that the historical intervention has a positive contribution to the current health improvement; when the attribution coefficient is negative, its corresponding attribution weight is used to allocate a negative reward share, indicating that the historical intervention has a negative contribution to the current health improvement. If the attribution coefficients of all historical monitoring periods and all health interventions within the preset traceability window are all zero, the system sets all attribution weights proportionally to zero, indicating that all historical interventions are unrelated to the current health indicator change and no reward allocation is required.

[0063] Furthermore, the reward for changes in health indicators in the current monitoring period is multiplied by the attribution weight corresponding to the health intervention program in the historical monitoring period. The resulting product is the cross-period allocated reward value obtained by the health intervention program in the historical monitoring period. The share of reward attributable to the specific intervention action in the historical monitoring period for the observed health improvement or deterioration in the current period is allocated according to the attribution weight corresponding to each intervention program in each historical period. This ensures that historical intervention actions with strong causal relationships receive a larger share of reward, while actions with weak or negative causal relationships receive a smaller share of reward or negative rewards, thus accurately reflecting the differentiated causal contribution of each historical intervention action to the changes in current health indicators. The above multiplication operation is performed independently for each historical monitoring period and each health intervention program within the preset traceability window to obtain their respective cross-period allocated reward values.

[0064] Furthermore, for each historical monitoring period, the cross-period allocated reward values ​​for all health intervention programs within that period are obtained, along with the immediate reward value given immediately after the intervention action is performed. All cross-period allocated reward values ​​and immediate reward values ​​for that historical monitoring period are summed; the resulting sum is the revised reward value for that historical monitoring period. This revised reward value includes not only the immediately observable feedback after the intervention action is performed in that period but also the delayed feedback share from changes in health indicators in subsequent periods, enabling the reinforcement learning model to comprehensively consider the long-term impact when evaluating the effectiveness of actions in that historical period.

[0065] Furthermore, based on the cumulative contribution of multiple options, the adaptive learning rate for the current update round is calculated according to the rule that the adaptive learning rate equals the standard learning rate multiplied by the maximum value between the cumulative contribution of multiple options and zero. When the cumulative contribution of multiple options is positive, the standard learning rate is multiplied by this positive value to obtain the adaptive learning rate. The larger the positive value, the larger the adaptive learning rate, indicating that the long-term net effect of historical intervention is good, accelerating the reinforcement of the current strategy. When the cumulative contribution of multiple options is zero or negative, the standard learning rate is multiplied by zero, and the adaptive learning rate is zero, indicating that the long-term net effect of historical intervention is zero or a negative obstacle. Strategy updates are paused to avoid strategy oscillation or deterioration due to negative experiences. The optimization objective is to maximize the cumulative correction reward value of all historical monitoring periods, and the adaptive learning rate is used to update the decision strategy execution parameters of the reinforcement learning model.

[0066] The construction of the reinforcement learning model includes: Multiple sets of historical health management cycle data are collected. Each set of data includes status data, action data, and corrected reward value for a historical monitoring cycle. The status data includes the current detection value of each health indicator and the periodic intervention execution volume of each health intervention plan. The action data is the intervention action executed in the historical monitoring cycle. The corrected reward value is the corrected reward value obtained after cross-cycle attribution based on the effect lag vector of the plan. The status data, action data, and corrected reward value constitute a training sample set. A reinforcement learning model is constructed, wherein the current detection value of each of the health indicators and the periodic intervention execution amount of each of the health intervention programs are used as state inputs, and the intervention action is used as output. The intervention action includes at least one of adjusting the intensity of exercise intervention, adjusting the dietary intervention program, and adjusting the medication intervention program. For each of the health indicators, the change in the periodic health indicator is used as the first reward component, and the product of the cumulative effect contribution of the multiple schemes and the change in the periodic health indicator is calculated as the second reward component. The sum of the first reward component and the second reward component is calculated and used as the instantaneous reward function of the health indicator. The sum of the instant reward function values ​​of all the health indicators is taken as the value of the instant reward function in the current monitoring period; For each historical monitoring period, the instant reward value of the historical monitoring period is first calculated according to the instant reward function, and then the health indicator change reward of the subsequent monitoring period is distributed in reverse along the time axis to the historical monitoring period to obtain the corrected reward value of the historical monitoring period. With the goal of maximizing the cumulative corrected reward value, the model parameters are updated, and the reinforcement learning model is trained using the training sample set until the preset convergence condition is met, thus obtaining the reinforcement learning model.

[0067] In this embodiment of the invention, during the construction and training phase of the reinforcement learning model, multiple sets of historical health management cycle data are first collected. Each set of data corresponds to a historical monitoring cycle and includes state data, action data, and adjusted reward values ​​for that cycle. The state data includes the current detection values ​​of each health indicator and the periodic intervention execution volume of each health intervention plan; the action data consists of the intervention actions output by the system decision within that historical monitoring cycle, including at least one of adjusting the intensity of exercise intervention, adjusting the dietary intervention plan, and adjusting the medication intervention plan; the adjusted reward value is the adjusted reward value obtained after cross-cycle attribution based on the time lag vector of the plan's effect. The above data from all historical cycles constitute the training sample set.

[0068] Furthermore, a reinforcement learning model is constructed. This model takes the current detection values ​​of each health indicator and the periodic intervention execution volume of each health intervention plan as state inputs, and the intervention action as output. Specifically, the reinforcement learning model adopts a policy gradient-based network architecture, which consists of two sub-networks: an action decision network and a state value evaluation network. The action decision network is responsible for outputting intervention action decisions based on the current state; the state value evaluation network is responsible for evaluating the long-term value of the current state, providing gradient direction guidance for the update of the action decision network.

[0069] The action decision network consists of an input layer, hidden layers, and an output layer connected sequentially. The input layer receives a state vector, which is composed of the current detection values ​​of each health indicator and the periodic intervention execution amounts of each health intervention plan. The number of neurons in the input layer equals the total dimension of the state vector. The hidden layers employ a multilayer perceptron structure, containing two to four fully connected hidden layers. Each hidden layer contains sixty-four to two hundred and fifty-six neurons, and each hidden layer is followed by a modified linear unit as the activation function to introduce nonlinear transformation capabilities. The output layer is designed according to the action space type. When the intervention action space is discrete, the output layer uses a flexible maximum activation function to output the probability distribution of each discrete action, sampling or selecting the action with the highest probability as the decision output. When the intervention action space is continuous, the output layer uses a linear activation function to directly output continuous action values.

[0070] The State Value Evaluation Network (SVE) shares the same state input as the Action Decision Network (ADN). Its hidden layer structure is identical to the ADN, and its output layer contains only one neuron, outputting the estimated value of the current state—the expected cumulative reward obtainable by executing the current policy in that state. The training objective of the SVE is to minimize the mean squared error between its output state estimate and the actually observed cumulative reward. During training, the state estimate output by the SVE is used to calculate the advantage function, which measures the superiority of performing a specific action relative to the average level under the current policy. This advantage function serves as the weight coefficient for the gradient update of the ADN, guiding the ADN to optimize in a direction that increases the probability of selecting high-value actions.

[0071] The parameter update of the action decision network adopts the proximal policy optimization algorithm. This algorithm calculates the probability ratio of the new policy and the old policy to take the same action in the same state, and restricts the ratio within a preset shear interval to constrain the update magnitude and avoid the policy performance from deteriorating drastically due to excessively large single update step size.

[0072] Specifically, for each health indicator, a separate immediate reward function is designed. The immediate reward function comprises two components. The first reward component is the periodic change in the health indicator itself, i.e., the improvement in the indicator compared to the previous period in the current monitoring period. This component directly reflects the short-term effect in the current period. The second reward component is the product of the cumulative contribution of the multiple solutions to the health indicator and the periodic change in the health indicator. This product amplifies the positive reward when the cumulative contribution is positive and imposes an additional penalty when it is negative, thus incorporating long-term contribution information into the immediate reward signal. The sum of the first and second reward components is the value of the immediate reward function for the health indicator in the current monitoring period. When multiple health indicators exist, the immediate reward function values ​​of all health indicators are summed to obtain the total immediate reward value for the current monitoring period.

[0073] Finally, for each historical monitoring period, the immediate reward value for that period is first calculated using the aforementioned immediate reward function. Then, the rewards for changes in health indicators in subsequent monitoring periods are distributed backward along the time axis to that historical monitoring period to obtain the corrected reward value for that historical monitoring period. The optimization objective is to maximize the cumulative corrected reward value of all training samples across historical monitoring periods. The reinforcement learning model is iteratively trained using the training sample set. After each training round, the change in the cumulative corrected reward value is calculated. When the cumulative corrected reward value no longer increases after several consecutive rounds, the model is deemed to have met the preset convergence condition, training is terminated, and the final reinforcement learning model is output.

[0074] This invention addresses the problem of attribution errors caused by neglecting delay effects in traditional instant reward mechanisms by allocating the reward for changes in health indicators in the current period to each historical monitoring period within a preset traceability window according to the attribution coefficient of each historical period. The allocated reward is then added to the instant reward of each historical period to obtain a corrected reward value. This enables the reinforcement learning model to accurately identify historical intervention actions that actually produce health improvement effects during policy updates. It significantly improves the causal logic and directional correctness of policy updates.

[0075] By dynamically calculating the adaptive learning rate based on the cumulative contribution of multiple strategies, the model accelerates positive reinforcement when the long-term net effect of historical strategies is good, and makes careful adjustments when the net effect is poor, effectively improving the model's convergence speed and training stability. By using the corrected reward value generated by the attribution result as the training sample label, and combining the immediate reward component and the long-term contribution component to construct the reward function, the reinforcement learning model can simultaneously consider the comprehensive evaluation of short-term effects and long-term contributions during training. The final output decision strategy can achieve accurate adaptive closed-loop health management based on the user's individual historical feedback.

[0076] In summary, the embodiments of the present invention have at least the following technical effects: This invention eliminates temporal misalignment by aligning execution records and health indicator data from different sources along the timeline and filling in missing data. It eliminates dimensional barriers between multiple intervention programs by dividing the actual number of executions by the required number. Furthermore, it removes individual static baselines by subtracting the current period's detection value from the previous period's detection value, significantly reducing the probability of spurious correlations caused by dimensional inconsistencies and baseline drift in subsequent analyses. By performing a full-range time-shift traversal on the paired combinations of each intervention program and health indicator and independently calculating the Pearson correlation coefficient at each time shift, the constructed program effect lag vector fully records the distribution of all delayed correlation strengths, fundamentally solving the problem of causal misjudgment caused by neglecting delayed effects in traditional contemporaneous analysis methods. By normalizing the rewards for changes in health indicators in the current period and distributing them to each historical monitoring period according to the attribution coefficient, the complete transmission and conservation of reward signals over time are ensured. This enables the reinforcement learning model to accurately identify historical intervention actions that actually produce effects, solving the attribution error caused by the neglect of delayed effects in traditional instant reward mechanisms. At the same time, by dynamically calculating the adaptive learning rate based on the cumulative effect contribution and constructing a reward function that takes into account both short-term effects and long-term contributions, the model's convergence speed and training stability are effectively improved. The final output decision strategy can achieve precise adaptive closed-loop health management based on individual user historical feedback.

[0077] In summary, this invention solves the technical problem in existing health management data analysis where inconsistent dimensions of multi-source data and time delays in intervention effects prevent reinforcement learning's immediate reward mechanism from accurately attributing contributions to historical actions.

[0078] Example 2, as Figure 2 As shown, based on the same inventive concept as the health management data analysis method combining reinforcement learning provided in Embodiment 1, this embodiment of the invention also provides a health management data analysis system combining reinforcement learning, including: Data acquisition module 11 is used to acquire users' historical execution records and time-series data of health indicators for various health intervention programs; The periodic quantification module 12 is used to calculate the periodic intervention execution amount of each health intervention program in each monitoring period and the periodic health indicator change amount of each health indicator. The time lag analysis module 13 is used to determine the time lag vector of the program effect of each health intervention program on each health indicator by performing cross-period time lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator. The cross-cycle attribution module 14 is used to perform cross-cycle effect attribution on the changes in the periodic health indicators of each monitoring cycle based on the time lag vector of the scheme effect, and obtain the cumulative effect contribution of multiple schemes. The reinforcement learning strategy update module 15 is used to update the decision-making strategy by distributing the reward for changes in health indicators in the current period in reverse along the time axis to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme effect, in the reward allocation of the reinforcement learning model.

[0079] In one embodiment, the data acquisition module 11 is specifically used for: Obtain the health intervention program execution records of the user in each monitoring cycle within a preset historical time period. Each execution record includes the intervention program type, the number of times it should be executed, and the number of times it is actually executed. The intervention program type includes at least two of the following: exercise intervention, dietary intervention, and medication intervention. The health indicator data of the user in each monitoring period within the preset historical time period are obtained, wherein the health indicator includes at least one of weight, blood sugar and blood pressure, and the health indicator data are arranged in chronological order according to the monitoring period to form health indicator time series data.

[0080] In one embodiment, the periodic quantization module 12 is specifically used for: Obtain the number of times each health intervention plan should be executed and the number of times it is actually executed in each monitoring period. Divide the number of times it is actually executed by the number of times it should be executed to obtain the periodic intervention execution amount of the health intervention plan in the monitoring period. Based on the time-series data of the health indicators, the detection value of each health indicator in the current monitoring period and the detection value in the previous monitoring period are obtained. The detection value in the previous monitoring period is subtracted from the detection value in the current monitoring period to obtain the periodic health indicator change in the current monitoring period.

[0081] In one embodiment, the time delay analysis module 13 is specifically used for: For each combination of the health intervention program and each health indicator, obtain the periodic intervention execution sequence of the program and the periodic health indicator change sequence of the health indicator, wherein the periodic intervention execution sequence and the periodic health indicator change sequence contain the same number of monitoring periods. The time shift range is set to multiple time shift values ​​between zero and a preset maximum time shift, wherein the preset maximum time shift is the longest delay period in which the health intervention plan may produce an observable effect on health indicators; For each time-shifted value, the sequence of changes in the periodic health indicators is shifted forward by the number of periods corresponding to the time-shifted value, so that the shifted sequence of changes in indicators is aligned period by period with the portion of the unshifted sequence of periodic intervention execution that does not exceed the shift range. Calculate the Pearson correlation coefficient of the aligned portion sequence, and use it as the time-delay correlation coefficient corresponding to the time-shift value; Arrange the time-lag correlation coefficients corresponding to all time-shift values ​​in ascending order of time-shift value to form the time-lag vector of the health intervention program's effect on the health indicator.

[0082] In one embodiment, the cross-period attribution module 14 is specifically used for: Obtain the periodic intervention execution volume of each health intervention plan in each historical monitoring period within a preset traceability window prior to the current monitoring period, wherein the length of the preset traceability window is not less than the preset maximum time shift; For each combination of the health intervention program and each health indicator, obtain the time lag vector of the program effect of the health intervention program on the health indicator; For each historical monitoring cycle within the preset traceability window, the number of cycles of intervention execution of the health intervention plan in the historical monitoring cycle is obtained, and the difference between the current monitoring cycle and the historical monitoring cycle is calculated to obtain the corresponding time lag cycle number. The time-shift value of the time-delay vector of the scheme effect is extracted as the time-delay correlation coefficient corresponding to the time-delay period number, and used as the attribution coefficient corresponding to the historical monitoring period; Multiply the periodic intervention execution amount by the attribution coefficient, and the resulting product is used as the attribution contribution component of the health intervention program to the health indicator within the historical monitoring period; The attribution contribution components of all historical monitoring periods and all health intervention programs within the preset traceability window are added together, and the sum is used as the cumulative effect contribution of the health indicator across multiple programs.

[0083] The effect time delay vector of the proposed scheme is a multi-dimensional vector, and the k-th element of the vector is the time delay correlation coefficient when the time shift value is k periods.

[0084] In one embodiment, the reinforcement learning policy update module 15 is specifically used for: Obtain the change in the health indicator during the current monitoring period, and use the change in the health indicator as a reward for the change in the health indicator during the current period. For each historical monitoring period within a preset traceability window prior to the current monitoring period, the intervention actions performed in the historical monitoring period are obtained, and the attribution weight of each health intervention program in the intervention actions of the historical monitoring period on the reward for the change of health indicators in the current monitoring period is determined according to the attribution coefficient. Multiply the reward for changes in the health indicators by the attribution weight to obtain the cross-period allocation reward value of the health intervention program within the historical monitoring period; The cross-cycle allocation reward value for each historical monitoring period within the preset traceability window is added to the instant reward value for each historical monitoring period to obtain the corrected reward value for each historical monitoring period. The adaptive learning rate for the current update round is calculated based on the standard learning rate of the reinforcement learning model and the cumulative effect contribution of the multiple schemes. With the goal of maximizing the cumulative corrected reward value, the decision strategy of the reinforcement learning model is updated using the adaptive learning rate.

[0085] The construction of the reinforcement learning model includes: Multiple sets of historical health management cycle data are collected. Each set of data includes status data, action data, and corrected reward value for a historical monitoring cycle. The status data includes the current detection value of each health indicator and the periodic intervention execution volume of each health intervention plan. The action data is the intervention action executed in the historical monitoring cycle. The corrected reward value is the corrected reward value obtained after cross-cycle attribution based on the effect lag vector of the plan. The status data, action data, and corrected reward value constitute a training sample set. A reinforcement learning model is constructed, wherein the current detection value of each of the health indicators and the periodic intervention execution amount of each of the health intervention programs are used as state inputs, and the intervention action is used as output. The intervention action includes at least one of adjusting the intensity of exercise intervention, adjusting the dietary intervention program, and adjusting the medication intervention program. For each of the health indicators, the change in the periodic health indicator is used as the first reward component, and the product of the cumulative effect contribution of the multiple schemes and the change in the periodic health indicator is calculated as the second reward component. The sum of the first reward component and the second reward component is calculated and used as the instantaneous reward function of the health indicator. The sum of the instant reward function values ​​of all the health indicators is taken as the value of the instant reward function in the current monitoring period; For each historical monitoring period, the instant reward value of the historical monitoring period is first calculated according to the instant reward function, and then the health indicator change reward of the subsequent monitoring period is distributed in reverse along the time axis to the historical monitoring period to obtain the corrected reward value of the historical monitoring period. With the goal of maximizing the cumulative corrected reward value, the model parameters are updated, and the reinforcement learning model is trained using the training sample set until the preset convergence condition is met, thus obtaining the reinforcement learning model.

[0086] Example 3, as Figure 3 As shown, this application embodiment provides a smart chip 200, including: Memory 210 is used to store computer program 211; The processor 220 is used to read and execute the computer program 211, thereby realizing a health management data analysis method that combines reinforcement learning.

[0087] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0089] This specification and accompanying drawings are merely illustrative examples of the invention and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its scope. Therefore, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is intended to include these modifications and modifications.

Claims

1. A health management data analysis method combining reinforcement learning, characterized in that, include: Obtain users' historical execution records and time-series data of health indicators across various health intervention programs; Calculate the periodic intervention execution volume of each health intervention program and the periodic health indicator change of each health indicator in each monitoring period; By conducting cross-period time lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator, the time lag vector of the program effect of each health intervention program on each health indicator is determined. Based on the time lag vector of the scheme effect, the cross-cycle effect attribution is performed on the change of the periodic health index in each monitoring period to obtain the cumulative effect contribution of multiple schemes. In the reward allocation of the reinforcement learning model, the reward for changes in health indicators in the current period is distributed in reverse along the time axis to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme's effect, and the decision-making strategy is updated.

2. The health management data analysis method combining reinforcement learning according to claim 1, characterized in that, The acquisition of users' historical execution records and time-series data of health indicators across multiple health intervention programs includes: Obtain the health intervention program execution records of the user in each monitoring cycle within a preset historical time period. Each execution record includes the intervention program type, the number of times it should be executed, and the number of times it is actually executed. The intervention program type includes at least two of the following: exercise intervention, dietary intervention, and medication intervention. The health indicator data of the user in each monitoring period within the preset historical time period are obtained, wherein the health indicator includes at least one of weight, blood sugar and blood pressure, and the health indicator data are arranged in chronological order according to the monitoring period to form health indicator time series data.

3. The health management data analysis method combining reinforcement learning according to claim 1, characterized in that, The calculation of the periodic intervention execution volume of each health intervention program and the periodic health indicator changes of each health indicator in each monitoring period includes: Obtain the number of times each health intervention plan should be executed and the number of times it is actually executed in each monitoring period. Divide the number of times it is actually executed by the number of times it should be executed to obtain the periodic intervention execution amount of the health intervention plan in the monitoring period. Based on the time-series data of the health indicators, the detection value of each health indicator in the current monitoring period and the detection value in the previous monitoring period are obtained. The detection value in the previous monitoring period is subtracted from the detection value in the current monitoring period to obtain the periodic health indicator change in the current monitoring period.

4. The health management data analysis method combining reinforcement learning according to claim 1, characterized in that, The step involves performing cross-period time-lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator to determine the time-lag vector of the program effect of each health intervention program on each health indicator, including: For each combination of the health intervention program and each health indicator, obtain the periodic intervention execution sequence of the program and the periodic health indicator change sequence of the health indicator, wherein the periodic intervention execution sequence and the periodic health indicator change sequence contain the same number of monitoring periods. The time shift range is set to multiple time shift values ​​between zero and a preset maximum time shift, wherein the preset maximum time shift is the longest delay period in which the health intervention plan may produce an observable effect on health indicators; For each time-shifted value, the sequence of changes in the periodic health indicators is shifted forward by the number of periods corresponding to the time-shifted value, so that the shifted sequence of changes in indicators is aligned period by period with the portion of the unshifted sequence of periodic intervention execution that does not exceed the shift range. Calculate the Pearson correlation coefficient of the aligned portion sequence, and use it as the time-delay correlation coefficient corresponding to the time-shift value; Arrange the time-lag correlation coefficients corresponding to all time-shift values ​​in ascending order of time-shift value to form the time-lag vector of the health intervention program's effect on the health indicator.

5. The health management data analysis method combining reinforcement learning according to claim 1, characterized in that, The method involves attributing the changes in periodic health indicators across monitoring periods based on the time lag vector of the proposed schemes, thereby obtaining the cumulative contribution of multiple schemes, including: Obtain the periodic intervention execution volume of each health intervention plan in each historical monitoring period within a preset traceability window prior to the current monitoring period, wherein the length of the preset traceability window is not less than the preset maximum time shift; For each combination of the health intervention program and each health indicator, obtain the time lag vector of the program effect of the health intervention program on the health indicator; For each historical monitoring cycle within the preset traceability window, the number of cycles of intervention execution of the health intervention plan in the historical monitoring cycle is obtained, and the difference between the current monitoring cycle and the historical monitoring cycle is calculated to obtain the corresponding time lag cycle number. The time-shift value of the time-delay vector of the scheme effect is extracted as the time-delay correlation coefficient corresponding to the time-delay period number, and used as the attribution coefficient corresponding to the historical monitoring period; Multiply the periodic intervention execution amount by the attribution coefficient, and the resulting product is used as the attribution contribution component of the health intervention program to the health indicator within the historical monitoring period; The attribution contribution components of all historical monitoring periods and all health intervention programs within the preset traceability window are added together, and the sum is used as the cumulative contribution of the health indicator to the multi-program effect.

6. The health management data analysis method combining reinforcement learning according to claim 5, characterized in that, The effect time delay vector of the proposed scheme is a multi-dimensional vector, and the k-th element of the vector is the time delay correlation coefficient when the time shift value is k periods.

7. The health management data analysis method combining reinforcement learning according to claim 1, characterized in that, In the reward allocation of the reinforcement learning model, the reward for changes in health indicators in the current period is distributed backward along the time axis to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme's effect, and the decision-making strategy is updated, including: Obtain the change in the health indicator during the current monitoring period, and use the change in the health indicator as a reward for the change in the health indicator during the current period. For each historical monitoring period within a preset traceability window prior to the current monitoring period, the intervention actions performed in the historical monitoring period are obtained, and the attribution weight of each health intervention program in the intervention actions of the historical monitoring period on the reward for the change of health indicators in the current monitoring period is determined according to the attribution coefficient. Multiply the reward for changes in the health indicators by the attribution weight to obtain the cross-period allocation reward value of the health intervention program within the historical monitoring period; The cross-cycle allocation reward value for each historical monitoring period within the preset traceability window is added to the instant reward value for each historical monitoring period to obtain the corrected reward value for each historical monitoring period. The adaptive learning rate for the current update round is calculated based on the standard learning rate of the reinforcement learning model and the cumulative effect contribution of the multiple schemes. With the goal of maximizing the cumulative corrected reward value, the decision strategy of the reinforcement learning model is updated using the adaptive learning rate.

8. The health management data analysis method combining reinforcement learning according to claim 7, characterized in that, The construction of the reinforcement learning model includes: Multiple sets of historical health management cycle data are collected. Each set of data includes status data, action data, and corrected reward value for a historical monitoring cycle. The status data includes the current detection value of each health indicator and the periodic intervention execution volume of each health intervention plan. The action data is the intervention action executed in the historical monitoring cycle. The corrected reward value is the corrected reward value obtained after cross-cycle attribution based on the effect lag vector of the plan. The status data, action data, and corrected reward value constitute a training sample set. A reinforcement learning model is constructed, wherein the current detection value of each of the health indicators and the periodic intervention execution amount of each of the health intervention programs are used as state inputs, and the intervention action is used as output. The intervention action includes at least one of adjusting the intensity of exercise intervention, adjusting the dietary intervention program, and adjusting the medication intervention program. For each of the health indicators, the change in the periodic health indicator is used as the first reward component, and the product of the cumulative effect contribution of the multiple schemes and the change in the periodic health indicator is calculated as the second reward component. The sum of the first reward component and the second reward component is calculated and used as the instantaneous reward function of the health indicator. The sum of the instant reward function values ​​of all the health indicators is taken as the value of the instant reward function in the current monitoring period; For each historical monitoring period, the instant reward value of the historical monitoring period is first calculated according to the instant reward function, and then the health indicator change reward of the subsequent monitoring period is distributed in reverse along the time axis to the historical monitoring period to obtain the corrected reward value of the historical monitoring period. With the goal of maximizing the cumulative corrected reward value, the model parameters are updated, and the reinforcement learning model is trained using the training sample set until the preset convergence condition is met, thus obtaining the reinforcement learning model.

9. A health management data analysis system incorporating reinforcement learning, characterized in that: A method for performing health management data analysis incorporating reinforcement learning as described in any one of claims 1 to 8, comprising: The data acquisition module is used to acquire users' historical execution records and time-series data of health indicators for various health intervention programs; The periodic quantification module is used to calculate the periodic intervention execution volume of each health intervention program and the periodic health indicator change volume of each health indicator in each monitoring period. The time lag analysis module is used to determine the time lag vector of the effect of each health intervention program on each health indicator by performing cross-period time lag correlation analysis on the periodic intervention execution sequence of each health intervention program and the periodic health indicator change sequence of each indicator. The cross-cycle attribution module is used to perform cross-cycle effect attribution on the changes in the periodic health indicators of each monitoring period based on the time lag vector of the scheme effect, and to obtain the cumulative effect contribution of multiple schemes. The reinforcement learning strategy update module is used to, in the reward allocation of the reinforcement learning model, revert the reward for changes in health indicators in the current period to the intervention actions in the historical periods that produced the effect, based on the time lag vector of the scheme's effect, and update the decision-making strategy.

10. A smart chip, characterized in that, The smart chip includes: Memory, used to store computer programs; A processor is configured to read and execute the computer program, thereby implementing the health management data analysis method incorporating reinforcement learning as described in any one of claims 1 to 8.

Images