Methods, systems, electronic devices, and storage media for motion timing push.

By constructing a current state vector and adjusting the reward value, the problem of insufficient user context representation in existing exercise reminder systems is solved, improving the accuracy and adaptability of exercise reminders. In particular, for new users or new situations, the model is trained using multi-user data to provide individualized intervention strategies.

CN123069074APending Publication Date: 2026-11-10BEIJING DONGLIANG HEALTH TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611287201.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-24
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing exercise reminder systems cannot fully represent the user's current situation, resulting in a mismatch between the timing of exercise reminders and the user's state. Furthermore, they lack sufficient decision-making basis for new users or new situations, affecting the accuracy and effectiveness of exercise reminder strategies.

Method used

By acquiring real-time contextual data of users to construct the current state vector, determine the target intervention action, and record the state-action record, the reward value is corrected based on the motion behavior detection results, the state-action value is updated using reinforcement learning, and a random forest classification model is trained in combination with multi-user historical interaction data to provide action selection for unexplored states.

Benefits of technology

It improves the adaptability and accuracy of the timing of exercise intervention, reduces the impact of delayed feedback on state-action value, and enhances the ability to form individualized intervention strategies for new users or new situations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069074A_ABST
    Figure CN123069074A_ABST
Patent Text Reader

Abstract

This invention discloses a method, system, electronic device, and storage medium for motion timing push notifications, relating to the interdisciplinary fields of behavioral strategies and reinforcement learning. The method acquires real-time user contextual data and constructs a current state vector. Based on the current state vector, it determines the target intervention action from push and non-push actions, forming a state-action record and adding it to a historical state-action sequence. It controls the output state of motion intervention information and detects target motion behavior based on the target intervention action. After detecting the target motion behavior, it determines the associated state-action record to be updated from the historical state-action sequence. It determines the reward value based on the intervention action, the motion behavior detection result, and the time interval between the action execution time and the motion behavior occurrence time, and updates the corresponding state-action value accordingly for subsequent intervention action determination. This invention can improve the adaptability and accuracy of motion intervention timing decisions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of behavioral strategy and reinforcement learning, and in particular to a method, system, electronic device and storage medium for pushing movement opportunities. Background Technology

[0002] With the development of mobile terminals, wearable devices, and health management applications, sending exercise reminders to users via electronic devices has become a common intervention method in sports and health management. Existing exercise reminder systems typically determine reminder timing based on pre-set times, cycles, or partial user status information. Some systems also use machine learning or reinforcement learning to gradually adjust exercise reminder strategies based on users' historical behavior and feedback. These methods can achieve a certain degree of automation and personalization in exercise reminders.

[0003] However, a user's location, physical activity, mobile device usage, and recent frequency of receiving exercise reminders can all change at different times, thus altering the appropriate timing for receiving such reminders. When existing exercise reminder methods utilize relatively limited state information, they fail to adequately represent the user's current situation, easily leading to a mismatch between the determined reminder timing and the user's current state, thereby affecting the effectiveness of subsequent reminder strategy adjustments.

[0004] On the other hand, after an exercise reminder is issued, users may not immediately begin exercising; there may be varying degrees of time delay between the actual exercise behavior and the previous reminder. When multiple exercise reminder decisions are made consecutively, it is not easy to accurately determine which previous decision is most strongly correlated with the subsequently detected exercise behavior. Insufficient consideration of the time relationship between user feedback and previous decisions can easily lead to inaccurate correspondence between feedback information and actual relevant decisions, thus affecting the accuracy of updating subsequent exercise reminder strategies based on user feedback.

[0005] Furthermore, when new users begin using the exercise reminder system, or when users enter situations they rarely encounter before, there is limited historical interaction data available for the current user. Existing exercise reminder methods that are gradually optimized based on personal historical data lack sufficient decision-making basis in these situations, often requiring multiple interactions to form a relatively stable personalized reminder strategy. This affects the effectiveness of the system in determining the timing of exercise reminders in the early stages of use or in new situations.

[0006] Therefore, how to further improve the ability to represent the user's current situation, improve the accuracy of the correspondence between the user's subsequent motion feedback and previous motion reminder decisions, and improve the effect of determining the timing of motion reminders when historical interaction data is insufficient are problems that existing motion timing push technologies still need to solve. Summary of the Invention

[0007] Based on the technical problems existing in the background art, the present invention proposes a method, system, electronic device and storage medium for pushing motion timing, which can improve the adaptability and accuracy of motion intervention timing decision-making.

[0008] The present invention proposes a motion timing push method, comprising: Acquire real-time contextual data of the user and construct the current state vector based on the real-time contextual data of the user. Based on the current state vector, the target intervention action is determined from the set of intervention actions, which includes pushing actions and non-pushing actions. The current state vector, the target intervention action and the corresponding action execution time are recorded to form a state-action record. The state-action record is then added to the historical state-action sequence. The system will push or not push motion intervention information based on the target intervention action, and obtain the detection results of the target motion behavior within the preset response time window; The state-action record to be updated is determined based on the detection results of the target motion behavior. When the detection result indicates that the target motion behavior has occurred, the state-action record associated with the target motion behavior is determined from the historical state-action sequence based on the time of occurrence of the target motion behavior as the state-action record to be updated. When the detection result indicates that the target motion behavior has not occurred, the state-action record corresponding to the current target intervention action is determined as the state-action record to be updated. The reward value corresponding to the action record to be updated is determined based on the intervention action and the detection results of the target motion behavior. When the detection results indicate that the target motion behavior has occurred, the reward value is corrected based on the time interval between the execution time of the action record to be updated and the occurrence time of the target motion behavior. The status-action value corresponding to the pending status-action record is updated based on the reward value, in order to determine subsequent intervention actions.

[0009] Preferably, the current state vector includes time state, location state, physical activity state, equipment usage state, emotional state, and intervention fatigue level; The time status is discretized and encoded according to the time of day; Location status is determined based on the mobile terminal's location information or preset environmental information; Body activity status is determined based on motion data collected by mobile terminals or wearable devices; The device's usage status is determined based on the interaction status of the mobile terminal; Emotional state is determined based on the acquired user emotional information; Intervention fatigue level is determined by weighting the push frequency, the proportion of users who do not respond, and the number of consecutive pushes within a preset historical period.

[0010] Preferably, determining the state-action records associated with the target motion behavior from the historical state-action sequence includes: Based on the time of occurrence of the target motion behavior, retrieve the historical state-action sequence in order of the action execution time from the most recent to the oldest; The state-action record that is executed before the occurrence of the target motion and has the smallest time interval between the two is identified as the state-action record to be updated.

[0011] Preferably, when the detection result characterizes the occurrence of target motion behavior, a positive basic reward is determined, and the reward value also includes a time correction amount determined based on the time interval; The test results indicate that if no target movement occurs within a preset non-response period after the target intervention action is executed, no basic reward is given, and a negative time correction amount is determined.

[0012] Preferably, the state-action value corresponding to the state-action record to be updated is updated according to the reward value, including: The temporal difference error is determined based on the reward value, the reward discount factor, the current state-action value corresponding to the state-action record to be updated, and the maximum value of the state-action value corresponding to each intervention action in the state at the next decision time corresponding to the state-action record to be updated. The state-action value corresponding to the state-action record to be updated is updated based on the temporal difference error. When the temporal difference error is positive, the state-action value corresponding to the state-action record to be updated is increased; when the temporal difference error is negative, the state-action value corresponding to the state-action record to be updated is decreased.

[0013] Preferably, the current state vector is determined to be either an explored state or an unexplored state based on its existence in the historical state set and the cumulative number of historical interactions corresponding to the current state vector. The cumulative number of historical interactions is the sum of the number of historical interactions corresponding to push actions and the number of historical interactions corresponding to no push actions under the current state vector; When the current state vector exists in the historical state set and the cumulative number of historical interactions reaches a preset value, the current state vector is determined as an explored state. Based on the state-action value of each intervention action corresponding to the current state vector, the intervention action with the highest state-action value is selected as the target intervention action. If the current state vector does not exist in the historical state set, or the cumulative number of historical interactions has not reached a preset value, the current state vector is determined as an unexplored state, and the current state vector is input into a state classification model trained based on multi-user historical interaction data to obtain the target intervention action.

[0014] Preferably, the state classification model is a random forest classification model; Acquire historical interaction data for multiple users. The historical interaction data includes state vectors, corresponding intervention actions, and target motion behavior detection results after the intervention actions are executed. For the same state vector, count the number of successful push actions and non-push actions respectively, and determine the action category label of the corresponding state vector based on the relationship between the number of successful push actions and non-push actions; A random forest classification model is trained using state vectors as input features and action category labels as output labels. When the current state vector is determined to be an unexplored state, the current state vector is input into the random forest classification model, and the predicted actions are output through multiple decision trees. The target intervention action is determined based on the predicted actions output by the multiple decision trees.

[0015] The present invention proposes a motion timing push system, comprising: The state construction module is used to acquire real-time contextual data of the user and construct the current state vector based on the real-time contextual data of the user. The intervention decision module is used to determine the target intervention action from the intervention action set based on the current state vector. The intervention action set includes push actions and non-push actions. It records the current state vector, the target intervention action and the corresponding action execution time to form a state-action record, and adds the state-action record to the historical state-action sequence. The output and detection module is used to push or not push motion intervention information according to the target intervention action, and to obtain the detection results of the target motion behavior within a preset response time window; The action attribution module is used to determine the state-action record to be updated based on the detection results of the target motion behavior. When the detection result indicates that the target motion behavior has occurred, the state-action record associated with the target motion behavior is determined from the historical state-action sequence based on the time of occurrence of the target motion behavior as the state-action record to be updated. When the detection result indicates that the target motion behavior has not occurred, the state-action record corresponding to the current target intervention action is determined as the state-action record to be updated. The reward determination module is used to determine the reward value corresponding to the state-action record to be updated based on the intervention action corresponding to the state-action record to be updated and the detection results of the target motion behavior. When the detection results indicate that the target motion behavior has occurred, the reward value is corrected according to the time interval between the execution time of the action in the state-action record to be updated and the occurrence time of the target motion behavior. The value update module is used to update the state-action value corresponding to the pending state-action record based on the reward value, so as to determine the subsequent intervention action.

[0016] The present invention provides an electronic device comprising: Processor; and Memory, which stores computer programs; When a computer program is executed by a processor, the processor performs the aforementioned motion timing push method.

[0017] The present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the aforementioned motion timing push method.

[0018] Beneficial technical effects of the present invention: (1) Each time a push action or no push action is determined, this application forms a state-action record by combining the current state vector, the intervention action, and the action execution time, and adds it to the historical state-action sequence. After detecting subsequent target motion behavior, the application then determines the state-action record associated with the motion behavior from the historical state-action sequence based on the occurrence time of the target motion behavior, and adjusts the reward value according to the time interval between the two. Thus, compared with the method of directly updating the reward of the current action after detecting motion behavior, it can use the time information of motion behavior occurrence to distinguish the degree of association between different historical intervention actions and motion behavior, reduce the situation where delayed motion feedback is used to update the state-action value with a low degree of association, thereby improving the pertinence of reward feedback and state-action value update, and providing more effective historical feedback information for determining the timing of subsequent motion intervention.

[0019] (2) This application constructs a current state vector using time state, location state, physical activity state, device usage state, emotional state, and intervention fatigue level, and establishes state-action value relationships between different states and push actions and non-push actions, so that the same user can correspond to different intervention action evaluation results in different time, location, activity, and device usage contexts. In particular, by incorporating intervention fatigue level, which is formed by recent push frequency, user non-response ratio, and consecutive push count, into the state vector, recent intervention history can be included in the current state representation, thereby reducing the insufficient timing matching caused by exercise reminders based solely on fixed push times or single state information, and improving the ability of exercise intervention actions to adjust to changes in the user's real-time situation.

[0020] (3) For unexplored states where the current user has not yet formed effective state-action value information, this application uses historical interaction data from multiple users to establish a state classification model, and forms training labels based on the motion response corresponding to pushing and not pushing actions in the same state; when the current state is determined to be an unexplored state, the state classification model outputs the target intervention action, while in states with sufficient historical feedback, the user's own state-action value is used to select actions. Thus, existing user interaction information can be used to provide a basis for action determination for new users or unexplored states, reducing the stage of no effective historical information caused by relying entirely on the current user to explore actions from the initial state, and facilitating the formation of individualized intervention strategies for the current user as user interaction data accumulates. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the functional modules of the motion timing push system proposed in this invention; Figure 2 This is a flowchart of the motion timing push method proposed in this invention; Figure 3 This is a schematic diagram illustrating the construction of the current state vector proposed in this invention; Figure 4 This is a flowchart illustrating the intervention action determination process for explored and unexplored states as proposed in this invention. Figure 5 This is a flowchart illustrating the action attribution and reward determination process for target motion behavior proposed in this invention. Figure 6 This is a schematic diagram of the historical state-action sequence and motion behavior attribution timeline proposed in this invention; Figure 7 This is a schematic diagram of the state-action value update based on temporal difference proposed in this invention. Detailed Implementation

[0022] The present invention will be further explained below with reference to specific embodiments.

[0023] Example 1 like Figure 2 As shown, the motion timing push method provided in this embodiment includes user real-time context data acquisition, current state vector construction, target intervention action determination, state-action record generation, target motion behavior detection, determination of state-action record to be updated, determination of reward value, and state-action value update.

[0024] First, acquire real-time user contextual data and construct a current state vector based on this data. Real-time user contextual data can be obtained from at least one of mobile terminals, wearable devices, and user input devices, and is used to characterize the user's current context at the moment of decision-making.

[0025] like Figure 3 As shown, in this embodiment, the current state vector includes time state T, location state L, physical activity state B, device usage state D, emotional state E, and intervention fatigue level F.

[0026] The time state T is used to characterize the time when the current decision occurs. In one specific implementation, it is discretized according to the time of day, with hours as the discrete time unit, then: T∈{0,1,2,…,23} Different values ​​correspond to different times of day.

[0027] Location status L is determined based on the mobile terminal's location information or user-preset environmental information. For example, location status can be categorized into home, workplace, and other location types.

[0028] Body activity state B is determined based on motion data collected by mobile terminal sensors or wearable devices. For example, body activity states can be categorized into sedentary, indoor, walking, running, and driving states based on the collected motion data.

[0029] The device usage state D is determined based on the interaction state of the mobile terminal. For example, the active and inactive states of the mobile terminal's screen are encoded as the corresponding device usage states.

[0030] Emotional state E is determined based on the acquired user emotional information. In one specific implementation, the user can actively input their current emotional information via a mobile terminal and categorize their emotional state into neutral, happy, excited, relaxed, angry, and anxious.

[0031] Intervention fatigue level F is used to reflect the frequency of recent exercise interventions received by the user and their response to previous exercise interventions. In one specific implementation, intervention fatigue level is determined according to the following formula: F = 0.5P + 0.3N + 0.2C Where P represents the push frequency in the last 7 days; N represents the proportion of users who did not respond in the last 7 days; C represents the number of consecutive pushes; P, N and C can all be determined based on the historical motion intervention records saved by the system.

[0032] Therefore, the current state vector corresponding to the current decision time t can be represented as: s t = (T, L, B, D, E, F) Among them, s t This represents the current state vector corresponding to decision time t.

[0033] After obtaining the current state vector, the target intervention action is determined based on the current state vector. The set of intervention actions A is represented as: A = {Push, No Push} "Push" means to output exercise intervention information to the user at the current decision-making moment, while "Do not push" means not to output exercise intervention information to the user at the current decision-making moment. "Do not push" is a valid action in the set of intervention actions and participates in subsequent state-action value recording and reward updates, and does not mean that the system has not executed the intervention decision.

[0034] like Figure 4 As shown, before determining the target intervention action, the current state can be divided into an explored state or an unexplored state based on the existence of the current state vector in the historical state set and the historical feedback of the state-action combination corresponding to the current state vector.

[0035] This embodiment uses the cumulative number of historical interactions corresponding to the current state vector as the criterion for determining whether the state has reached the explored condition. For the current state s... t The cumulative number of historical interactions is determined according to the following formula: N(s t )=N(s t (push) + N(s) t (No push notifications) Wherein, N(s) t ) represents the current state s t The corresponding cumulative number of historical interactions; N(s) t N(s) represents the number of historical interactions corresponding to the push action in the current state; t (No push) indicates the number of historical interactions corresponding to actions that are not pushed in the current state. It is not required that push actions and non-push actions each reach a preset number of times.

[0036] When the current state vector exists in the historical state set, and N(s) tWhen a preset value is reached, the current state is defined as an explored state; otherwise, the current state is defined as an unexplored state. In one specific implementation, the preset value can be 20 times.

[0037] For the explored states, obtain the state-action value corresponding to each intervention action under the current state vector, and determine the intervention action with the highest state-action value as the target intervention action, that is: Among them, a t Q(s) represents the target intervention action determined at decision time t; t a) represents the current state s t The state-action value corresponding to the intervention action 'a' is determined below.

[0038] For example, if the current state s t If the state-action value corresponding to the push action is greater than the state-action value corresponding to the no-push action, then the push action is identified as the target intervention action; otherwise, the no-push action is identified as the target intervention action.

[0039] For unexplored states, the current state vector is input into a state classification model trained based on historical interaction data from multiple users to obtain the target intervention action.

[0040] In this embodiment, the state classification model adopts the random forest classification model.

[0041] When training the random forest classification model, we first acquire historical interaction data from multiple users during the exercise intervention process. Each historical interaction data entry includes at least a state vector, the corresponding intervention action, and the target motion behavior detection result after the intervention action is executed. The random forest classification model uses s t As input features.

[0042] For the same state vector, the number of successes after executing the push action and the number of successes after executing the no-push action are counted separately. A successful intervention is defined as detecting the user's target movement behavior within 30 minutes after the intervention action is executed.

[0043] The action category label for each status is determined based on the relationship between the number of successful push notifications and non-push notifications. The label determination relationship is as follows: Where Y represents the action category label corresponding to this state. Represents state s t The number of successful actions for different actions in the action set A.

[0044] When the number of successful push actions is higher than the number of successful non-push actions under the same state, the action category label for that state is determined to be push; when the number of successful non-push actions is higher than the number of successful push actions, the action category label for that state is determined to be non-push.

[0045] This generates training data with the current state vector as input features and push or no push action as output labels. The training data is then used to train multiple decision trees in the random forest classification model, enabling each decision tree to learn the correspondence between state features and action categories.

[0046] When the current state vector is determined to be an unexplored state, the current state vector s is... t Input a trained random forest classification model, from which multiple decision trees output predicted actions (Tree). i (s t ), where i represents the number of the corresponding decision tree.

[0047] The final target intervention action is determined by majority voting based on the prediction results of multiple decision trees, which can be represented as: a t =Vote(Tree1(s t ),Tree2(s t ),...,Tree m (s t )) Where m represents the number of decision trees in the random forest classification model, and Vote represents the majority vote for the prediction actions of multiple decision trees.

[0048] Therefore, when the current user already has historical feedback data for the corresponding state, the current user's own state-action value can be used to select actions; while for new users or when the current user enters an unexplored state, the motion intervention experience formed from the historical interaction data of multiple users can be used to provide an initial basis for action determination.

[0049] As users continue to use the system, the number of successful pushes and non-pushes can be updated based on the user's actual movement feedback in the corresponding state.

[0050] When the user actually reports that the exercise was successful after the push notification: Success(s t (push) += 1; When the user's actual feedback is that the exercise failed after the push notification: Success(s t (push) += 0; When the user's actual feedback is that the exercise was successful after no push notification: Success(s) t(No push notifications) += 1; When the user's actual feedback is that the exercise failed after the push notification was not sent: Success(s t (No push notification) += 0.

[0051] By using the above methods, historical interaction data can be continuously added to the user's actual exercise feedback, so as to gradually accumulate exercise intervention data corresponding to the user's current state.

[0052] After determining the target intervention action, regardless of whether the intervention action is a push or not, a corresponding state-action record is generated. The state-action record includes at least the current state vector, the target intervention action, and the corresponding action execution time. In one specific implementation, the state-action value Q value corresponding to the current state-action combination can also be recorded.

[0053] Multiple state-action records are stored according to the execution time of their respective actions, thus forming a historical state-action sequence.

[0054] For example, if the target intervention action is determined to be push at 09:00, a first state-action record is formed; if the intervention decision is executed again at 10:00 and the target intervention action is determined to be no push, a second state-action record is formed. The first state-action record and the second state-action record are added to the historical state-action sequence in sequence according to the time of action execution.

[0055] Based on the determined target intervention action, the system will either push or not push motion intervention information, and obtain the detection results of the target motion behavior within the preset response time window.

[0056] In one specific implementation, the preset response time window is set to the same day after the current intervention decision occurs. After the system completes the decision to push or not push, it continuously monitors whether the user has engaged in the target movement behavior within this response time window.

[0057] Targeted exercise behavior can be identified through at least one of the following methods: the user clicks the "Start Exercise" button in a fitness app; the wearable device detects that the user has entered a continuous walking, running, or cycling state; the user manually enters exercise data such as exercise type and duration through a mobile terminal. As long as any of the above conditions are met, it can be determined that the user has engaged in a targeted exercise behavior.

[0058] Based on the target intervention action and the target motion behavior detection results, four response scenarios can be identified: target motion behavior occurs after push notification, target motion behavior does not occur after push notification, target motion behavior occurs after no push notification, and target motion behavior does not occur after no push notification.

[0059] When a target motion behavior is detected, the state-action record associated with the target motion behavior is determined from the historical state-action sequence based on the time when the target motion behavior occurs, and this record is used as the state-action record to be updated.

[0060] like Figure 5 and Figure 6 As shown, this embodiment uses a recent action priority method based on time relationship to determine the state-action record to be updated.

[0061] This embodiment adopts a reward attribution method based on time windows and recent action priority.

[0062] Each time the system makes a push or no push decision, it creates a state-action record containing the current state vector, action type, action execution time, and corresponding Q value. After detecting target motion behavior, the system retrieves historical state-action sequences in ascending order of action execution time, based on the time the target motion behavior occurred.

[0063] For multiple state-action records located before the time when the target motion occurs, the state-action record with the smallest time interval between the time when the target motion occurs is identified as the state-action record to be updated that is associated with the current target motion.

[0064] For example, if a push action is performed at 09:00, a no-push action is performed at 10:00, and target motion behavior is detected at 10:10, then the time interval between the push action at 09:00 and the target motion behavior is 70 minutes, and the time interval between the no-push action at 10:00 and the target motion behavior is 10 minutes. Therefore, according to the principle of prioritizing the most recent action, the state-action record corresponding to the no-push action at 10:00 is determined as the state-action record to be updated.

[0065] In this way, when multiple intervention decisions are made in succession, the reward corresponding to a single target movement behavior is not simultaneously distributed to all historical state-action records. Instead, the state-action records that are more temporally related to the target movement behavior are updated first.

[0066] When no target motion behavior is detected within the preset response time window, the state-action record formed by the target intervention action corresponding to this response detection is determined as the state-action record to be updated.

[0067] After determining the status-action record to be updated, the reward value is determined based on the intervention action corresponding to the status-action record to be updated and the target motion behavior detection results.

[0068] In one specific implementation, upon detecting the target's motion behavior, the final reward value R can be expressed as: R=R b +R t +R f Among them, R b This represents the base reward determined based on the user's motion response when the target motion behavior is detected; R t R represents the reward adjustment value determined based on the motor response time after the target intervention action is performed. f This indicates the intervention fatigue correction index.

[0069] When target motion behavior is detected, R t The value of R is determined based on the time difference between the intervention execution time and the user's actual movement participation time; if no target movement behavior is detected within 120 minutes after the action execution, R is determined as a non-response situation. t ; In one specific implementation, when target motion is detected within the response time window: R b =5; If no target motor behavior is detected within 120 minutes after the target intervention action is performed, no basic reward R is assigned because no motor response occurs. b The reward adjustment value corresponding to this non-response is determined as: R t =−4.

[0070] If the target motion behavior is detected, the basic reward is further dynamically adjusted based on the time difference between the time of push or no push action and the time of the user's actual motion participation.

[0071] When a user engages in a target movement behavior within 0 to 30 minutes after a push notification or no push notification action is executed: R t =2; When a user engages in the target movement behavior within 30–60 minutes after a push or no push action is executed: R t =−1; When a user engages in the target movement behavior within 60–120 minutes after a push or no push action is executed: R t =−2; When the user does not participate in exercise within 120 minutes after the push or no push action is executed: R t =−4, used to give a larger negative evaluation of the corresponding state-action.

[0072] For the intervention fatigue correction index R f First, the intervention fatigue level F is calculated using the aforementioned formula. When the intervention fatigue level exceeds a set threshold, i.e., F > F... threshold Time: R f =−1; otherwise: R f =0.

[0073] Therefore, the reward feedback corresponding to the current state-action is formed by combining whether the user has performed the target movement behavior, the time relationship between the target movement behavior and the intervention action, and the current level of fatigue during the intervention.

[0074] In this embodiment, examples of rewards for different states and actions are shown in Table 1.

[0075] Table 1 Examples of Rewards for Different States and Actions Through the aforementioned reward mechanism, the system can evaluate push and non-push actions in different real-time situations based on the user's actual movement feedback. In particular, when the user still performs the target movement behavior relatively quickly without a push notification, positive feedback can be generated for the corresponding non-push status-action, enabling subsequent decisions to identify whether additional movement reminders are needed in this state.

[0076] After the reward is determined, combined with Figure 7 Update the status and action value corresponding to the pending status and action record based on the reward value.

[0077] This embodiment employs a reinforcement learning update method based on temporal difference learning, combined with a qualification trace mechanism to update the state-action value Q.

[0078] After the system completes an intervention decision and obtains actual user motion feedback, it first calculates the temporal difference error corresponding to the state-action to be updated. The formula for the temporal difference error is: Among them, s t Indicates the current state; a t Indicates the current action; r represents the reward value corresponding to this action; s t+1 Indicates the state at the next moment; Indicates the current state-action value; γ represents the maximum Q value among all actions in the next state; γ represents the reward discount factor.

[0079] In one specific implementation: γ = 0.9 Temporal difference error is used to represent the difference between the target value formed based on the current user's actual feedback and the existing state-action value.

[0080] When the time-series difference error δ is greater than 0, it indicates that the actual feedback corresponding to the current intervention action is better than the result reflected by the model, thus increasing the value of the state-action. When the time-series difference error δ is less than 0, it indicates that the actual feedback corresponding to the current intervention action is lower than the result reflected by the model, thus decreasing the value of the state-action.

[0081] Furthermore, in this embodiment, the qualification trace is used as the qualification weight corresponding to the state-action.

[0082] The qualification trace state formula is as follows: record'(s t ,a t = record(s) t ,a t )×γ Where, record(s) t ,a t ) represents the current state s t and action a t The corresponding qualification weight. record'(s t ,a t ) represents the updated state s t and action a t Corresponding qualification weight In one specific implementation, the eligibility weight corresponding to each state-action is initially set to 0.1. When a user experiences a movement event within 30 minutes of executing the current action, the eligibility weight for the corresponding state-action is updated according to the above formula; otherwise, the eligibility weight update is not performed. The reward discount factor γ is set to 0.9.

[0083] Combining the time-series difference error δ and the eligibility weight, the Q-value update formula is: Q'(s t ,at)=Q(s t ,a t )+δ×record'(s t ,a t ) Where Q(s) t (,at) indicates that in state s t Next, execute action a t The corresponding state-action value, Q'(s) t (,at) indicates that after the update, in state s t Next, execute action a t The corresponding state-action value; the initial value of each state-action value is set to 0; δ represents the aforementioned temporal difference error.

[0084] After updating the state-action value, the updated state-action value is saved to the current user's historical data. When the user subsequently enters a state with sufficient historical feedback, the next targeted intervention action is determined based on the Q-values ​​corresponding to the updated push action and the no-push action.

[0085] Therefore, as users continue to use the system, the process of repeatedly acquiring real-time user context data, constructing current state vectors, determining target intervention actions, generating state-action records, detecting target motion behavior, attributing historical state-actions, determining rewards, and updating state-action values ​​enables users' subsequent real-time motion behaviors to continuously generate feedback information for different real-time contexts and different intervention actions.

[0086] Through the above implementation process, on the one hand, the user's current immediate situation is described using time state, location state, physical activity state, equipment usage state, emotional state, and intervention fatigue level, so that the same user can form different intervention action evaluation results in different immediate situations; on the other hand, when there is a time delay between the user's actual movement behavior and the previous intervention action, the state-action record with a high degree of correlation is determined from the historical state-action sequence based on the time when the target movement behavior occurs, so that the corresponding movement feedback can be used to update the corresponding historical state-action value; for unexplored states where the current user's own historical data is insufficient, the target intervention action is determined using a random forest classification model trained based on multi-user historical interaction data, and corresponding individual historical information is gradually formed as the current user's own data accumulates.

[0087] Example 2 like Figure 1 As shown, this embodiment provides a motion timing push system, including a state construction module, an intervention decision module, an output and detection module, an action attribution module, a reward determination module, and a value update module.

[0088] The state construction module is used to acquire real-time contextual data of the user and construct the current state vector based on the real-time contextual data of the user.

[0089] Real-time user contextual data can be obtained through at least one of mobile terminals, wearable devices, and user-initiated input. The current state vector includes time state, location state, physical activity state, device usage state, emotional state, and intervention fatigue level. The determination of each state and the calculation of intervention fatigue level can adopt the method in Example 1.

[0090] The intervention decision module receives the current state vector generated by the state construction module and determines the target intervention action from the push action and non-push action based on the current state vector.

[0091] When the current state is an explored state, the intervention decision module reads the state-action value corresponding to the push action and the non-push action in the current state, and determines the target intervention action based on the corresponding state-action value.

[0092] When the current state is an unexplored state, the intervention decision module calls the random forest classification model trained based on the historical interaction data of multiple users, inputs the current state vector into the random forest classification model, and determines the target intervention action based on the prediction results of multiple decision trees in the random forest classification model.

[0093] The intervention decision module is also used to record the current state vector, the target intervention action, and the corresponding action execution time, forming a state-action record, and adding the formed state-action record to the historical state-action sequence. In one specific embodiment, the state-action record also includes the state-action value Q value corresponding to the current state and corresponding action.

[0094] The output and detection module is used to push or not push motion intervention information based on the target intervention action, and to obtain the detection results of the target motion behavior within a preset response time window.

[0095] When the target intervention action is a push action, the output and detection module controls the corresponding output terminal to output motion intervention information to the user; when the target intervention action is a non-push action, no corresponding motion intervention information is output at the current decision moment, but the state-action record formed by the current decision is still saved.

[0096] The output and detection module is also used to obtain subsequent user exercise behavior data. Exercise behavior data can come from at least one of the following: start exercise information entered by the user in the exercise application, continuous walking, running or cycling information detected by the wearable device, and exercise type and duration information actively recorded by the user.

[0097] The action attribution module is used to determine the state to be updated - the action record - based on the target motion behavior detection results.

[0098] When the detection result indicates that a target motion behavior has occurred, the action attribution module searches the historical state-action sequence from the nearest to the furthest in terms of the action execution time, based on the time when the target motion behavior occurred. The state-action record that is located before the time when the target motion behavior occurred and has the smallest time interval with the time when the target motion behavior occurred is identified as the state-action record to be updated.

[0099] When no target motion behavior is detected within the preset response time window, the state-action record formed by the target intervention action corresponding to the current response detection is determined as the state-action record to be updated.

[0100] The reward determination module is used to determine the corresponding reward value based on the intervention action corresponding to the status-action record to be updated and the target motion behavior detection results.

[0101] When a target motion behavior is detected, the reward determination module also corrects the reward based on the time relationship between the execution time of the action corresponding to the state-action record to be updated and the time when the target motion behavior occurs; in the implementation method that adopts intervention fatigue correction, the reward is also corrected according to the current intervention fatigue level.

[0102] The value update module is used to update the state-action value corresponding to the state-action record to be updated based on the reward value determined by the reward determination module.

[0103] In one specific implementation, the value update module uses the time-series difference error calculation, qualification weight update and Q-value update method of Example 1 to update the Q-value corresponding to the state-action record to be updated, and saves the updated Q-value to the historical data corresponding to the current user, so that the intervention decision module can call it when determining the target intervention action in the future.

[0104] In a specific system implementation, the above functions can be achieved using a sensing module, a storage module, and an output module.

[0105] The sensing module is used to acquire user health status and real-time contextual data, which can be obtained through wearable devices, smartphone sensors, and user input.

[0106] The storage module stores user data and learning strategies, and deploys reinforcement learning model units, event-driven reward backtracking units, and transfer learning units. Each unit can interact with the perception module and the output module.

[0107] The reinforcement learning model unit is used to determine whether to push or not to push an action based on the user's real-time contextual state; the event-driven reward backtracking unit is used to determine the corresponding reward based on the user's actual movement behavior and update the historical state-action value; the transfer learning unit, in this embodiment, provides action prediction for the current user's unexplored state through a state classification model trained on multi-user historical interaction data.

[0108] according to Figure 1 The functional divisions shown are as follows: the corresponding functions of the state construction module can be implemented by the perception module and its associated data processing functions; the corresponding functions of the intervention decision module can be implemented by the reinforcement learning model unit; the corresponding functions of the action attribution module, reward determination module, and value update module can be implemented by the event-driven reward backtracking unit; the random forest classification function corresponding to the unexplored state can be implemented by the state classifier in the transfer learning unit; and the corresponding functions of the output and detection module can be implemented by the output module and the mobile terminal or wearable device that interacts with it.

[0109] The modules described above are divided according to their data processing functions. Each module can be implemented by a corresponding data processing program, and it is not required that each functional module correspond to independently configured physical hardware.

[0110] Example 3 This embodiment provides an electronic device.

[0111] The electronic device includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it causes the processor to execute the motion timing push method of Embodiment 1.

[0112] Specifically, when the processor executes the computer program, it acquires the user's real-time contextual data and constructs a current state vector based on the user's real-time contextual data; it determines the target intervention action from the set of intervention actions, including push actions and non-push actions, based on the current state vector, and records the current state vector, the target intervention action and the corresponding action execution time to form a state-action record, and adds the state-action record to the historical state-action sequence.

[0113] The processor further performs data processing to control whether or not to push motion intervention information based on the target intervention action, and obtains the detection results of the target motion behavior within a preset response time window.

[0114] When a target motion behavior is detected, the state-action record associated with the target motion behavior is determined from the historical state-action sequence as the state-action record to be updated, based on the time when the target motion behavior occurs; when no target motion behavior is detected, the state-action record formed by the target intervention action corresponding to this response detection is determined as the state-action record to be updated.

[0115] The processor determines the reward value based on the intervention action corresponding to the state-action record to be updated and the target motion behavior detection result. When the target motion behavior is detected, the reward value is corrected according to the time relationship between the execution time of the corresponding action and the occurrence time of the target motion behavior. Then, the state-action value corresponding to the state-action record to be updated is updated according to the reward value for the determination of subsequent intervention actions.

[0116] The processor can also determine whether the current state is an explored state or an unexplored state based on the existence of the current state vector in the historical state set and the corresponding historical feedback. For explored states, the target intervention action is determined based on the state-action value corresponding to different intervention actions in the current state. For unexplored states, the target intervention action is determined by calling a random forest classification model trained on historical interaction data of multiple users.

[0117] In addition to storing computer programs, the memory can also store real-time user context data, current state vector, historical state set, historical state-action sequence, state-action value corresponding to different state-action combinations, multi-user historical interaction data, and state classification model related data generated during the execution of the above-mentioned motion timing push method.

[0118] Thus, by executing the computer program stored in the memory through the processor, the electronic device is able to implement the motion timing push method of Embodiment 1.

[0119] Example 4 This embodiment provides a computer-readable storage medium.

[0120] A computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the motion timing push method of Embodiment 1.

[0121] Specifically, when a computer program is executed by a processor, the processor acquires real-time contextual data of the user and constructs a current state vector based on the user's real-time contextual data. Based on the current state vector, the target intervention action is determined from push actions and non-push actions, and the current state vector, the target intervention action, and the time of action execution are recorded to form a state-action record. The formed state-action record is then added to the historical state-action sequence.

[0122] The computer program also enables the processor to push or not push motion intervention information based on the target intervention action, and to obtain the detection results of the target motion behavior within a preset response time window; to determine the state-action record to be updated based on the target motion behavior detection results; to determine the reward value based on the intervention action corresponding to the state-action record to be updated and the target motion behavior detection results; when the target motion behavior is detected, to correct the reward value based on the time relationship between the execution time of the action in the state-action record to be updated and the occurrence time of the target motion behavior; and to update the state-action value corresponding to the state-action record to be updated based on the reward value for use in determining subsequent intervention actions.

[0123] The computer program can also enable the processor to perform actions such as determining the explored and unexplored states of the current state, invoking the random forest classification model in the unexplored state, attributing the target's motion behavior to historical state-action records, determining rewards, and updating state-action value based on temporal differences.

[0124] Therefore, when an electronic device with data processing capabilities reads and executes a computer program stored in a computer-readable storage medium, the above-mentioned motion timing push method can be implemented.

Claims

1. A method for pushing motion timing, characterized in that, include: Acquire real-time contextual data of the user and construct the current state vector based on the real-time contextual data of the user. Based on the current state vector, the target intervention action is determined from the set of intervention actions, which includes pushing actions and non-pushing actions. The current state vector, the target intervention action and the corresponding action execution time are recorded to form a state-action record. The state-action record is then added to the historical state-action sequence. The system will push or not push motion intervention information based on the target intervention action, and obtain the detection results of the target motion behavior within the preset response time window; The state-action record to be updated is determined based on the detection results of the target motion behavior. When the detection result indicates that the target motion behavior has occurred, the state-action record associated with the target motion behavior is determined from the historical state-action sequence based on the time of occurrence of the target motion behavior as the state-action record to be updated. When the detection result indicates that the target motion behavior has not occurred, the state-action record corresponding to the current target intervention action is determined as the state-action record to be updated. The reward value corresponding to the action record to be updated is determined based on the intervention action and the detection results of the target motion behavior. When the detection results indicate that the target motion behavior has occurred, the reward value is corrected based on the time interval between the execution time of the action record to be updated and the occurrence time of the target motion behavior. The status-action value corresponding to the pending status-action record is updated based on the reward value, in order to determine subsequent intervention actions.

2. The motion timing push method according to claim 1, characterized in that, The current state vector includes time state, location state, physical activity state, equipment usage state, emotional state, and intervention fatigue level; The time status is discretized and encoded according to the time of day; Location status is determined based on the mobile terminal's location information or preset environmental information; Body activity status is determined based on motion data collected by mobile terminals or wearable devices; The device's usage status is determined based on the interaction status of the mobile terminal; Emotional state is determined based on the acquired user emotional information; Intervention fatigue level is determined by weighting the push frequency, the proportion of users who do not respond, and the number of consecutive pushes within a preset historical period.

3. The motion timing push method according to claim 1, characterized in that, Identify state-action records associated with the target's motion behavior from historical state-action sequences, including: Based on the time of occurrence of the target motion behavior, retrieve the historical state-action sequence in order of the action execution time from the most recent to the oldest; The state-action record that is executed before the occurrence of the target motion and has the smallest time interval between the two is identified as the state-action record to be updated.

4. The motion timing push method according to claim 3, characterized in that, When the detection results characterize the target motion behavior, a positive basic reward is determined, and the reward value also includes a time correction amount determined based on the time interval; The test results indicate that if no target movement occurs within a preset non-response period after the target intervention action is executed, no basic reward is given, and a negative time correction amount is determined.

5. The motion timing push method according to claim 1, characterized in that, Update the status and action value corresponding to the pending status and action record based on the reward value, including: The temporal difference error is determined based on the reward value, the reward discount factor, the current state-action value corresponding to the state-action record to be updated, and the maximum value of the state-action value corresponding to each intervention action in the state at the next decision time corresponding to the state-action record to be updated. The state-action value corresponding to the state-action record to be updated is updated based on the temporal difference error. When the temporal difference error is positive, the state-action value corresponding to the state-action record to be updated is increased; when the temporal difference error is negative, the state-action value corresponding to the state-action record to be updated is decreased.

6. The motion timing push method according to claim 1, characterized in that, Based on the existence of the current state vector in the historical state set and the cumulative number of historical interactions corresponding to the current state vector, determine whether the current state vector is an explored state or an unexplored state. The cumulative number of historical interactions is the sum of the number of historical interactions corresponding to push actions and the number of historical interactions corresponding to no push actions under the current state vector; When the current state vector exists in the historical state set and the cumulative number of historical interactions reaches a preset value, the current state vector is determined as an explored state. Based on the state-action value of each intervention action corresponding to the current state vector, the intervention action with the highest state-action value is selected as the target intervention action. If the current state vector does not exist in the historical state set, or the cumulative number of historical interactions has not reached a preset value, the current state vector is determined as an unexplored state, and the current state vector is input into a state classification model trained based on multi-user historical interaction data to obtain the target intervention action.

7. The motion timing push method according to claim 6, characterized in that, The state classification model is a random forest classification model; Acquire historical interaction data for multiple users. The historical interaction data includes state vectors, corresponding intervention actions, and target motion behavior detection results after the intervention actions are executed. For the same state vector, count the number of successful push actions and non-push actions respectively, and determine the action category label of the corresponding state vector based on the relationship between the number of successful push actions and non-push actions; A random forest classification model is trained using state vectors as input features and action category labels as output labels. When the current state vector is determined to be an unexplored state, the current state vector is input into the random forest classification model, and the predicted actions are output through multiple decision trees. The target intervention action is determined based on the predicted actions output by the multiple decision trees.

8. A motion timing push system, characterized in that, include: The state construction module is used to acquire real-time contextual data of the user and construct the current state vector based on the real-time contextual data of the user. The intervention decision module is used to determine the target intervention action from the intervention action set based on the current state vector. The intervention action set includes push actions and non-push actions. It records the current state vector, the target intervention action and the corresponding action execution time to form a state-action record, and adds the state-action record to the historical state-action sequence. The output and detection module is used to push or not push motion intervention information according to the target intervention action, and to obtain the detection results of the target motion behavior within a preset response time window; The action attribution module is used to determine the state-action record to be updated based on the detection results of the target motion behavior. When the detection result indicates that the target motion behavior has occurred, the state-action record associated with the target motion behavior is determined from the historical state-action sequence based on the time of occurrence of the target motion behavior as the state-action record to be updated. When the detection result indicates that the target motion behavior has not occurred, the state-action record corresponding to the current target intervention action is determined as the state-action record to be updated. The reward determination module is used to determine the reward value corresponding to the state-action record to be updated based on the intervention action corresponding to the state-action record to be updated and the detection results of the target motion behavior. When the detection results indicate that the target motion behavior has occurred, the reward value is corrected according to the time interval between the execution time of the action in the state-action record to be updated and the occurrence time of the target motion behavior. The value update module is used to update the state-action value corresponding to the pending state-action record based on the reward value, so as to determine the subsequent intervention action.

9. An electronic device, characterized in that, include: processor; as well as Memory, which stores computer programs; When a computer program is executed by a processor, the processor performs the motion timing push method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the motion timing push method according to any one of claims 1-7.