Pillow adaptive adjustment method and system based on reinforcement learning

Abstract

The invention relates to the technical field of smart home, and discloses a pillow adaptive adjustment method and system based on reinforcement learning, and the method comprises the steps: constructing a safety constraint manifold representation model, and mapping a physiological safety boundary into a manifold boundary; constructing a Riemannian measurement model, and fusing multi-objective optimization and security constraint information; realizing a strategy optimization algorithm based on Riemannian measurement; a self-adaptive Pareto optimal point switching mechanism is realized; according to the method, unified representation and processing of safety constraint and multi-objective optimization are realized through the constraint manifold reinforcement learning framework, and on the premise of ensuring physiological safety, the comfort level, health support and sleep quality of the user are efficiently balanced and optimized, so that the user experience is improved. The safety, the multi-target balance capability, the optimization efficiency and the self-adaptive adjustment capability of the pillow adjustment system are improved, and personalized, safe and efficient intelligent pillow adjustment services are provided for users.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and more specifically, to a pillow adaptive adjustment method and system based on reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and smart home technology, smart pillows have received widespread attention as an important device for improving sleep quality. Traditional fixed-shape pillows cannot dynamically adjust according to the user's sleeping posture and physiological state, easily leading to problems such as cervical discomfort and decreased sleep quality. Although some adjustable smart pillow products have appeared on the market, their adjustment methods are still mainly based on simple preset modes or basic feedback control, which is difficult to meet the personalized sleep needs of users.

[0003] In terms of control algorithms for smart pillows, existing technologies mainly have three key problems: lack of effective processing capability for multi-objective optimization, making it difficult to simultaneously consider multiple objectives such as user comfort, cervical spine health, and sleep quality; inadequacy in handling safety constraints, with existing methods often treating safety constraints simply as penalty terms or fixed boundaries, making it difficult to guarantee the safety of the adjustment process; and the separation of safety constraints from multi-objective optimization leads to complex algorithm structures and low optimization efficiency.

[0004] While reinforcement learning has made significant progress in adaptive control, its application in smart pillow adjustment systems still presents significant challenges in effectively handling safety constraints and multi-objective optimization. Especially when multiple objectives and multiple safety constraints need to be considered simultaneously, traditional reinforcement learning methods often require the construction of complex constraint processing modules and objective balancing algorithms, which not only increases algorithmic complexity but also easily leads to optimization getting trapped in local optima. Summary of the Invention

[0005] This invention provides a pillow adaptive adjustment method and system based on reinforcement learning, which solves the technical problem in related technologies that it is difficult to achieve efficient and unified optimization of multiple objectives (such as comfort, health support and sleep quality) while meeting physiological safety constraints.

[0006] This invention provides a pillow adaptive adjustment method based on reinforcement learning, comprising the following steps: Construct a safety-constrained manifold representation model, including collecting user information and physiological parameters, defining a set of constraint functions based on the physiological model, and defining the safety-constrained manifold as a subset of the state-action space that satisfies the constraint conditions; Based on the safety-constrained manifold, a Riemannian metric model is constructed, a set of multi-objective value functions is defined, and a Riemannian metric tensor that integrates multi-objective gradients and safety-constraint information is built to realize a dynamic weight adjustment mechanism. Using the constructed Riemannian metric model, a policy optimization algorithm based on Riemannian metric is implemented. A natural policy gradient method on the manifold is constructed, and the safety constraints are ensured through policy parameter updates and manifold constraint projection. Based on the optimized policy parameters, an adaptive Pareto optimal point switching mechanism is implemented, a Pareto front representation model is constructed, the optimal point is selected based on the sleep state, and a smooth transition mechanism is used to ensure parameter changes.

[0007] In a preferred embodiment, the step of constructing the security-constrained manifold representation model includes: Collect user information and physiological parameters; The set of constraint functions is defined based on the physiological model, including cervical spine pressure constraint function, head support angle constraint function, and neck muscle tension constraint function. Define a safety-constrained manifold as a subset of the state-action space that satisfies the constraints. Implement a manifold representation algorithm to determine in real time whether a state-action pair is within a safe manifold.

[0008] In a preferred embodiment, the step of constructing the Riemann metric model includes: Define a set of multi-objective value functions, including comfort value functions, health support value functions, and sleep quality value functions; Construct a Riemannian metric tensor that integrates multi-objective gradient and security constraint information; Implement a dynamic weight adjustment mechanism to adjust the weights of different objectives based on the user's sleep stage and feedback; Calculate the inverse matrix of the Riemann metric, which is used for policy optimization calculations.

[0009] In a preferred embodiment, the construction of the Riemann metric tensor includes the following steps: Calculate the gradient of the value function for each optimization objective; Calculate the gradient for each constraint function; Based on the target weight and constraint weight parameters, the target gradient and constraint gradient are weighted and combined to construct the Riemann metric tensor.

[0010] In a preferred embodiment, the steps of implementing the policy optimization algorithm based on Riemann metric include: Constructing natural policy gradient methods on manifolds; Implement the strategy parameter update algorithm; Safety constraints are ensured to be met through policy parameter updates and manifold constraint projection. To implement a reinforcement learning training process based on experience playback.

[0011] In a preferred embodiment, the strategy parameter update process includes: Calculate the gradient of the policy loss function; The gradient is transformed using the inverse matrix of the Riemann metric; The parameters are updated based on the learning rate.

[0012] In a preferred embodiment, manifold constraint projection is achieved through the following steps: Calculate the distance between the updated parameters and the safe manifold; An iterative projection method is used to project the parameters onto the nearest safe manifold point.

[0013] In a preferred embodiment, the steps of implementing the adaptive Pareto optimality switching mechanism include: Construct a Pareto front representation model; Select the optimal point based on sleep state; A smooth transition mechanism ensures that parameter changes are handled correctly. Implement a Pareto front adaptive update algorithm based on user feedback.

[0014] In a preferred embodiment, the smooth transition mechanism is implemented through the following steps: The smoothing coefficient is dynamically adjusted based on the current sleep state and depth. Based on the policy parameters from the previous time step and the policy parameters corresponding to the target Pareto optimal point, linear interpolation is performed using a smoothing coefficient to obtain the actual policy parameters to be executed.

[0015] In a preferred embodiment, a reinforcement learning-based adaptive pillow adjustment system is used to execute a reinforcement learning-based adaptive pillow adjustment method, comprising: The data acquisition module includes a smart pillow actuator, a sensor array system, an edge computing processing unit, and a communication interface module, used to collect user sleep data and physiological parameters; The data processing module includes functions for sensor data preprocessing, state feature extraction, sleep stage identification, and physiological indicator evaluation, which are used to analyze and process the collected data. The reinforcement learning module includes constraint function set, Riemann metric construction, multi-objective weight adjustment, manifold boundary detection, policy optimization, and constraint projection functions, which are used to achieve multi-objective optimization under safe constraints. The control execution module includes functions such as user interface interaction, adjustment command execution, user feedback collection, and sleep report generation, which are used to realize the human-computer interaction and control execution of the system.

[0016] The beneficial effects of this invention are as follows: By constructing a constrained manifold representation model and a Riemannian metric model, a unified representation and processing of safety constraints and multi-objective optimization is achieved. This method can efficiently balance and optimize multiple objectives such as user comfort, health support, and sleep quality while ensuring physiological safety constraints such as cervical spine pressure and head support angle, significantly improving the safety and multi-objective balancing ability of pillow adjustment systems.

[0017] A policy optimization algorithm based on Riemann metric is adopted, which improves the convergence speed and optimization efficiency by searching for the optimal policy on a constrained manifold. Meanwhile, the manifold constraint projection mechanism ensures that safety constraints are always met during the reinforcement learning exploration process, effectively avoiding user discomfort caused by inappropriate adjustments.

[0018] The adaptive Pareto optimal switching mechanism dynamically adjusts the optimization focus based on the user's sleep stage and physiological feedback, providing personalized adjustment services. Through a smooth transition control algorithm, the system can smoothly switch between different Pareto optimal points, avoiding abrupt adjustments that could disrupt the user's sleep.

[0019] This invention employs a layered architecture design, modularly encapsulating functions such as hardware control, data processing, constraint representation, reinforcement learning optimization, and application interaction, thereby improving the system's scalability and maintainability. Through the deployment of edge computing units, the system can achieve real-time response and localized decision-making, enhancing the timeliness and reliability of adjustments. Attached Figure Description

[0020] Figure 1 This is a flowchart of a pillow adaptive adjustment method based on reinforcement learning according to the present invention; Figure 2 This is the health support-comfort balance curve of the present invention; Figure 3 This is a bar chart comparing the safety indicators of different methods of the present invention; Figure 4 This is a bar chart comparing the optimization efficiency of different methods of the present invention; Figure 5 This is a bar chart comparing the coverage area of ​​the Pareto front in this invention; Figure 6 This is a radar chart comparing the user experience index scores of this invention; Figure 7 This is a bar chart comparing the learning efficiency of different methods of the present invention. Detailed Implementation

[0021] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0022] At least one embodiment of the present invention discloses a pillow adaptive adjustment method based on reinforcement learning, such as... Figure 1 As shown, it includes the following steps: Step 1: Construct a safety-constrained manifold representation model, including collecting user information and physiological parameters, defining a set of constraint functions based on the physiological model, and defining the safety-constrained manifold as a subset of the state-action space that satisfies the constraint conditions; Specifically, the following steps are included: Step 1.1: Collect basic user information and physiological parameters, including basic physiological parameters such as height, weight, age, cervical curvature, and neck length, as well as any special physiological conditions the user may have (such as a history of cervical spondylosis, abnormal cervical curvature, etc.).

[0023] Step 1.2: Define a set of safety constraint functions based on the physiological model; In this embodiment, the set of safety constraint functions includes: Cervical spine pressure constraint function Ensure that the pressure at each node of the cervical spine does not exceed the safe threshold; Head support angle constraint function Ensure that the head support angle is within a reasonable range; Neck muscle tension constraint function Avoid maintaining a posture that leads to muscle fatigue for extended periods; in, It indicates the system status, including the user's current posture, physiological indicators, etc. This indicates adjustment actions, including adjustment parameters such as pillow height and firmness.

[0024] Specifically, these constraint functions are implemented as follows: Cervical spine pressure constraint function: ; in, This represents the cervical spine pressure constraint function, used to determine whether the cervical spine pressure is within a safe range; It indicates the current state of the system, including information such as user posture and physiological indicators; This indicates adjustment actions, including adjustment parameters such as pillow height and firmness; Indicates the state Next action Predicted cervical spine pressure values ​​after surgery; This represents the maximum safe stress threshold calculated based on the user's physiological characteristics (such as age, weight, etc.); when When the predicted pressure does not exceed the safety threshold, it meets the cervical spine pressure safety constraint.

[0025] The calculation process for the cervical spine pressure constraint function is as follows: Based on the current state and candidate actions, the pressure distribution of each node in the cervical spine is predicted using a biomechanical model to obtain the predicted pressure value. A personalized maximum safe pressure threshold is calculated based on the user's physiological characteristics (such as age, weight, cervical spine condition, etc.). The constraint function value is obtained by subtracting the maximum safety threshold from the predicted pressure value.

[0026] When this value is less than or equal to zero, it indicates that the predicted pressure does not exceed the safety threshold and meets the safety constraints. When this value is greater than zero, it indicates that the predicted pressure exceeds the safety threshold, violating safety constraints.

[0027] Head support angle constraint function: ; in, This represents the head support angle constraint function, used to determine whether the head support angle is within a safe range; It indicates the current state of the system, including information such as user posture and physiological indicators; This indicates adjustment actions, including adjustment parameters such as pillow height and firmness; Indicates the state Next action Predicted head support angle; This indicates the maximum safe support angle calculated based on the user's physiological characteristics (such as cervical curvature, age, etc.). This represents the minimum safe support angle calculated based on the user's physiological characteristics; when When the predicted angle is within a safe range, it satisfies the head support angle safety constraint.

[0028] The calculation process for the head support angle constraint function is as follows: Predict the head support angle based on the current state and candidate actions; Determine the upper and lower limits of the safe range for the head support angle based on the user's physiological characteristics; Calculate two differences: one is the predicted angle minus the maximum safe angle, and the other is the minimum safe angle minus the predicted angle. Take the larger of the two differences as the constraint function value.

[0029] When this value is less than or equal to zero, it indicates that the predicted angle is within the safe range and meets the safety constraints. When this value is greater than zero, it indicates that the predicted angle exceeds the safe range and violates the safety constraints.

[0030] Neck muscle tension constraint function: ; in, This represents the neck muscle tension constraint function, used to determine whether the neck muscle tension is within a safe range; It indicates the current state of the system, including information such as user posture and physiological indicators; This indicates adjustment actions, including adjustment parameters such as pillow height and firmness; Indicates the state Next action Predicted values ​​of neck muscle tension after surgery; This represents the maximum safe tension threshold calculated based on the user's physiological characteristics (such as age, weight, etc.) and current sleep duration; when When the predicted tension does not exceed the safety threshold, it satisfies the neck muscle tension safety constraint.

[0031] The calculation process for the neck muscle tension constraint function is as follows: Based on the current state and candidate actions, the tension value of the neck muscles is predicted using a biomechanical model; The maximum safe tension threshold is calculated based on the user's physiological characteristics and current sleep duration. This threshold decreases as sleep time increases, reflecting the physiological phenomenon that maintaining the same posture for a long time may lead to muscle fatigue. The constraint function value is obtained by subtracting the maximum safety threshold from the predicted tension value.

[0032] When this value is less than or equal to zero, it indicates that the predicted tension does not exceed the safety threshold and meets the safety constraints. When this value is greater than zero, it indicates that the predicted tension exceeds the safety threshold, violating the safety constraints.

[0033] Step 1.3: Construct a mathematical model of the safety-constrained manifold; A safety-constrained manifold is defined as a subset of the state-action space that satisfies all safety constraints. ; in, Let represent a safety-constrained manifold, which is the set of combinations of states and actions that satisfy all safety constraints. It indicates the current state of the system, including information such as user posture and physiological indicators; This indicates adjustment actions, including adjustment parameters such as pillow height and firmness; The state space represents the set of all possible system states. The action space represents the set of all possible adjustment actions; Indicates the first The first safety constraint function is used to determine whether the state-action combination satisfies the first... One safety constraint; This represents the total number of security constraint functions, in this embodiment. These correspond to three constraints: cervical spine pressure, head support angle, and neck muscle tension, respectively. This represents the Cartesian product of the state-action pair belonging to the state space and the action space. Indicates the first The value of each constraint function is less than or equal to 0, which means that the safety constraint is satisfied. This means that the condition must be satisfied for all constraint functions.

[0034] Step 1.4: Implement the manifold representation algorithm to determine in real time whether a state-action pair is located within a safe manifold; The algorithm receives the current state. and candidate actions As input, calculate the values ​​of all constraint functions and determine whether they are all satisfied. The conditions are used to determine Does it belong to a safe manifold?

[0035] like Figure 2 As shown, this diagram illustrates the ability of three methods—constrained manifold RL, traditional multi-objective RL, and single-objective RL with constraints—to balance the two objectives of comfort and health support. The horizontal axis represents the direction of increasing comfort, and the vertical axis represents the health support index. The curve for the constrained manifold RL method is significantly higher than the other two methods, indicating that it provides better health support at the same level of comfort.

[0036] Step 2: Based on the safety-constrained manifold, construct the Riemann metric model, define a set of multi-objective value functions, construct a Riemann metric tensor that integrates multi-objective gradients and safety constraint information, and realize a dynamic weight adjustment mechanism; Specifically, the following steps are included: Step 2.1, define multiple objective value functions, corresponding to different optimization objectives: Comfort value function, used to evaluate the impact of pillow adjustment parameters on user comfort; Health support value function: evaluates the support effect of pillow adjustment parameters on cervical spine health; Sleep quality value function, used to assess the impact of pillow adjustment parameters on sleep quality; These value functions are implemented using deep neural networks, with the state as the input. and actions The output is the expected value of the corresponding target.

[0037] Specifically, these deep neural networks adopt the following structure: Input layer: Receives the state vector (Including user posture, physiological indicators, and other features) and motion vectors (A spliced ​​vector containing adjustment parameters such as pillow height and firmness); Feature extraction layer: consists of 3 fully connected layers, each containing 128, 64, and 32 neurons respectively, using the ReLU activation function; Attention-weighted layer: Dynamically weights the importance of different physiological signal features, enabling the model to focus on corresponding key features according to different sleep stages; Output layer: A single neuron that outputs the expected score of the corresponding target.

[0038] Each value function network is trained using a combination of supervised learning and reinforcement learning. Initial training uses expert-labeled data for supervised pre-training, followed by further optimization using real user interaction data and reinforcement learning.

[0039] In some embodiments, the deep neural network structure described above can be replaced with a lighter model, such as a 2-layer fully connected network (64-32 neurons) structure, which is suitable for edge devices with limited computing resources.

[0040] In addition, according to another embodiment of this application, the attention weighting layer can be omitted, and a fully connected layer can be used directly for feature fusion, which simplifies the model structure but may slightly reduce performance.

[0041] Step 2.2: Construct the Riemann metric tensor and fuse multi-objective gradients and security constraint information: ; in, This represents the Riemannian metric tensor, used to measure the "distance" in different directions in the parameter space; It indicates the current state of the system, including information such as user posture and physiological indicators; This indicates adjustment actions, including adjustment parameters such as pillow height and firmness; Indicates the first The weight coefficients of each optimization objective can be dynamically adjusted based on user preferences and the current sleep stage; Indicates the first A target value function, such as comfort, health support, etc.; The weight parameter represents the safety constraint and is used to balance the importance of the optimization objective and the safety constraint. Indicates the first Safety constraint functions, such as cervical spine pressure constraint and head support angle constraint; Indicates the first Partial derivative operator with one parameter; Indicates the first Partial derivative operator with one parameter; This represents the total number of optimization objectives; The total number of safety constraint functions.

[0042] In the above formula, and Represent the objective value function respectively Regarding the first and the The gradient of each parameter reflects the degree and direction of the influence of parameter changes on the target value; and Representing constraint functions respectively Regarding the first and the The gradient of each parameter reflects the degree and direction of the influence of parameter changes on the constraints; all gradients are calculated by automatic differentiation techniques, without the need to manually derive the partial derivative expressions.

[0043] Step 2.3: Implement a dynamic weight adjustment method to adjust the weights of different targets based on the user's sleep stage and feedback. ; For example, increase the weight of comfort during the user's sleep onset stage and increase the weight of health support during the deep sleep stage.

[0044] Step 2.4, calculate the inverse matrix of the Riemannian metric. This is used for subsequent strategy optimization calculations; Since the Riemannian metric tensor can be computationally complex in high-dimensional space, this implementation adopts an approximate calculation method, using the conjugate gradient method to solve the linear equation system, thus avoiding direct calculation of the inverse matrix.

[0045] like Figure 3 As shown, the safety violation probabilities of the three methods at different sleep stages are illustrated. The constrained manifold RL method has a significantly lower safety violation probability at all sleep stages than the other two methods, with an average safety violation probability of only 0.4%, while the traditional multi-objective RL and single-objective RL+ constraints are 3.7% and 2.5%, respectively.

[0046] Step 3: Using the constructed Riemannian metric model, implement the policy optimization algorithm based on Riemannian metric, construct the natural policy gradient method on the manifold, and ensure that the safety constraints are met through policy parameter updates and manifold constraint projection. Specifically, the following steps are included: Step 3.1: Construct the natural policy gradient method on the manifold; The formula for calculating the gradient of the natural policy is as follows: ; in, The natural policy gradient represents the optimal policy update direction in the parameter space. This represents the policy performance objective function, used to evaluate the overall performance of the current policy, and includes a weighted combination of multiple optimization objectives; Represents policy parameters, which include all trainable parameters in a policy network (such as a deep neural network); The Fisher information matrix is ​​used as a Riemannian metric in the parameter space to measure the "distance" in different directions within the parameter space. The policy objective function is expressed with respect to the parameters. The first derivative of , i.e., the gradient of the ordinary policy; This represents the inverse of the Fisher information matrix, used to convert ordinary gradients into natural gradients.

[0047] In this embodiment, the aforementioned constructed Riemannian metric is used. This replaces the standard Fisher information matrix, incorporating security constraints and multi-objective information into gradient computation. This substitution allows policy updates to simultaneously consider the effects of multiple optimization objectives and security constraints.

[0048] Step 3.2: Implement the strategy parameter update algorithm; The policy parameter update formula based on Riemann metric is: ; in, This represents the policy parameters for the next time step, which include all trainable weights and biases in the policy network; Indicates the policy parameters at the current moment; This represents the learning rate, which controls the step size for parameter updates. Indicates the current parameter The Riemannian metric inverse matrix at point is used to convert the ordinary gradient into the natural gradient; Indicates the policy parameters Operators for calculating gradients; This represents the policy loss function, which is a weighted combination of multiple optimization objectives (such as comfort, health support, etc.).

[0049] Step 3.3: Construct a manifold constraint projection operator to ensure that the updated parameters still generate actions that satisfy the safety constraints; The projection operator is defined as: ; in, This indicates projecting the parameters onto the safe manifold. Projection operator on; Indicates the strategy parameters; Indicates the update amount of the strategy parameters; Indicates the parameters of the new strategy after projection; This represents the safety-constrained manifold, i.e., the parameter space that satisfies all safety constraints; In Riemannian metric The distance metric function is as follows; This represents the parameter value that minimizes the objective function.

[0050] In practice, an iterative projection method is used to gradually adjust the parameters to the point closest to the safe zone.

[0051] In practice, the manifold constraint projection operator executes the following steps: Calculate the current policy parameters Does the generated action satisfy the safety constraints?

[0052] For a given state Through policy network Generate Actions .

[0053] Inspection action Do all safety constraints meet? .

[0054] If all constraints are met, return directly. As the updated parameter.

[0055] If a constraint violation exists, perform the following iterative projection process: Initialize the penalty coefficient vector for violating constraints.

[0056] Constructing the Lagrange function: ; in, This represents the Lagrangian function, used to transform constrained optimization problems into unconstrained optimization problems; ; Indicates the strategy parameters to be optimized; This represents the Lagrange multiplier vector, used to penalize violations of constraints; In Riemannian metric The distance metric function is as follows; This represents the original updated parameter point; This indicates summing over all constraints; Indicates the first Lagrange multipliers corresponding to each constraint; This represents the function that takes the maximum value. Indicates the first One constraint function; Indicates the current state of the system; This indicates that when the parameter is At that time, the policy network deals with the state The output action distribution.

[0057] Iterative optimization using gradient descent method Minimize the Lagrange function: ; in, Indicates the first The strategy parameters after the next iteration; Indicates the first The strategy parameters for the next iteration; Indicates the first The learning rate of each iteration is used to control the step size of parameter updates; Indicates the parameter Operators for calculating gradients; This indicates that the Lagrange function in the parameters and Lagrange multipliers The value at that location.

[0058] Update the Lagrange multipliers: ; in, Indicates the first The updated Lagrange multipliers corresponding to each constraint; Indicates the first Each constraint corresponds to the Lagrange multiplier before the update; This represents the step size parameter, which controls the update magnitude of the Lagrange multipliers; Indicates the first A constraint function in state and actions The value to be taken below; This represents a function that maximizes the value, ensuring that penalties are only incurred when constraints are violated. The parameter is The policy network in state The actions generated below .

[0059] Repeat steps c and d until convergence or the maximum number of iterations is reached.

[0060] It should be noted that the iterative projection method described above may be inefficient when computational resources are limited. Therefore, in some embodiments, this application also provides a simplified projection method that maps the constraint-violating actions onto the safety constraint boundary by directly truncating or scaling them. Although this method may not be as accurate as the iterative projection method, it is more computationally efficient and suitable for scenarios with high real-time requirements.

[0061] By using this iterative projection method, the system can maintain the gradient information of the original update direction as much as possible while ensuring safety constraints, thus avoiding the problems of low optimization efficiency or insufficient safety that may be caused by simple truncation or penalty term methods.

[0062] Step 3.4: Implement the reinforcement learning training process based on experience playback; The system stores user interaction data (status, action, reward, next state) and uses this data to periodically update the policy network parameters.

[0063] The entire training process follows these steps: Sample batch data from the experience buffer; Calculate multi-objective weighted rewards; The parameter update direction is calculated using the natural policy gradient method; Apply manifold constraint projection to ensure that the updated parameters still satisfy the safety constraints; Update the policy network parameters.

[0064] like Figure 4 As shown, the convergence times of the three methods are illustrated in different test scenarios. The average convergence time of the constrained manifold RL method is only 5.2 seconds, while the traditional multi-objective RL and single-objective RL+constraint require 13.8 seconds and 10.3 seconds, respectively, indicating that the optimization efficiency of the method in this invention is significantly improved.

[0065] Step 4: Based on the optimized policy parameters, implement an adaptive Pareto optimal point switching mechanism, construct a Pareto front representation model, select the optimal point based on the sleep state, and ensure parameter changes through a smooth transition mechanism; Specifically, the following steps are included: Step 4.1: Construct the Pareto front representation model; On a safety-constrained manifold, by changing the objective weight vector, a Pareto optimal solution set is generated and represented as a curve or surface on the manifold.

[0066] The system uses parameterized representation. ,in These are the positional parameters on the Pareto front, each Corresponding to a set of target weights and corresponding strategy parameters .

[0067] Step 4.2: Implement the Pareto point selection algorithm based on sleep state; The algorithm selects the Pareto optimum that best suits the current sleep stage (such as before falling asleep, light sleep, deep sleep, REM sleep, etc.) and physiological indicators.

[0068] The specific algorithm is as follows: Receive current sleep state and historical physiological data As input; Based on sleep state, identify current priority goals (such as prioritizing comfort before falling asleep, and prioritizing health support during deep sleep). Calculate the target weight vector in the current state. ; Find the optimal point closest to the current weight on the Pareto front. ; Output the corresponding strategy parameters .

[0069] Wherein, the target weight vector The calculation function is defined as follows: ; in, This represents the target weight vector at time t, used to determine the relative importance of different optimization objectives (such as comfort, health support, etc.). It indicates the sleep state at time t, such as the stage before falling asleep, light sleep, deep sleep, etc. This represents historical physiological data up to time t, including the user's physiological indicators and feedback records; Indicates based on sleep state The basic weight mapping function maps different sleep stages to corresponding target weight configurations, such as prioritizing comfort during the sleep onset stage. Indicated based on historical physiological data The weight adjustment function personalizes the base weights based on the user's historical feedback, reflecting the user's personal preferences. This represents the weighting coefficient for personalized adjustments, with a value range of [0, 1]. It increases as the system gains a better understanding of the user and is used to balance the proportion of basic configuration and personalized adjustments.

[0070] The process of calculating the target weight vector is as follows: Based on the currently detected sleep state (such as before falling asleep, light sleep, deep sleep, etc.), the basic target weight configuration is determined through a preset mapping relationship. For example, comfort is given priority during the fall asleep stage, and health support is given priority during the deep sleep stage. Based on users' historical physiological data and feedback, a personalized weight adjustment is calculated, which reflects the user's personal preferences. The personalized adjustment amount is multiplied by a weighting coefficient, which increases as the system's understanding of the user increases, enabling the system to gradually adapt to the user's personalized needs. The base weights are added to the weighted personalized adjustments to obtain the final target weight vector, which is used to select the optimal point on the Pareto front.

[0071] like Figure 5 As shown, the Pareto front coverage area exponents of the three methods are illustrated. The coverage area exponent of the constrained manifold RL method reaches 0.86, which is significantly higher than that of the traditional multi-objective RL (0.71) and single-objective RL+constraint (0.62), indicating that the method of this invention can provide a wider optimal solution space.

[0072] Step 4.3: Build a smooth transition module to ensure that the pillow parameters change gradually when switching to the Pareto optimal point, and avoid sudden changes that may interrupt the user's sleep. The smooth transition algorithm uses the following calculation method: ; in, The parameters representing the actual strategy implemented are the final parameter values ​​calculated through a smooth transition. This represents the smoothing coefficient, with a value range of [0, 1]. It is dynamically adjusted according to the current sleep depth; the deeper the sleep, the smoother the smoothing. The smaller the value, the smoother the parameter transition; This represents the strategy parameters at the previous moment, i.e., the parameter values ​​used in the previous execution. This represents the policy parameters corresponding to the target Pareto optimal point, i.e., the target parameter values ​​that the system expects to achieve. This represents the weighting coefficient of the parameters at the previous time step, which complements the smoothing coefficient to ensure that the weight sum is 1.

[0073] The specific implementation process of the smooth transition algorithm is as follows: Obtain the policy parameters used in the previous moment and the policy parameters corresponding to the target Pareto optimal point; The smoothing coefficient value is determined based on the current sleep state, and this coefficient reflects the rate of parameter change; Calculate the weighted average of the two sets of parameters, where the weight of the parameter at the previous time step is... Smoothing coefficient The weights of the target parameters are smoothing coefficients; The final weighted average is used as the strategy parameter for practical applications.

[0074] In this way, the system can ensure a smooth transition to the target parameters while avoiding sudden changes in parameters, thereby reducing interference with the user's sleep.

[0075] like Figure 6 As shown, the performance of the three methods on four user experience metrics—comfort, support, sleep quality, and overall satisfaction—is illustrated. The constrained manifold RL method scores higher than the other two methods on all metrics, especially achieving a score of 9.1 (out of 10) on the support metric.

[0076] Step 4.4: Implement the Pareto front adaptive update algorithm based on user feedback; The algorithm analyzes long-term user feedback and sleep quality data to adjust the Pareto frontier model, making it more suitable for the needs of specific users.

[0077] The update process includes: Collect user feedback data, including subjective evaluations and objective sleep indicators; Identify the Pareto region of user preferences; Add more sampling points in the vicinity of this area to improve the accuracy of the Pareto front in this region; Update the target weight mapping function to make the system more inclined to select the Pareto optimal point within the user's preferred region.

[0078] like Figure 7 As shown, the amount of interaction data required by the three methods at different learning stages is illustrated. The constrained manifold RL method requires less interaction data at all learning stages than the other two methods, totaling only 17.7 thousand interactions, while traditional multi-objective RL and single-objective RL+constraint require 62.8 thousand and 49.5 thousand interactions, respectively, indicating that the learning efficiency of the method of this invention is significantly improved.

[0079] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A pillow adaptive adjustment method based on reinforcement learning, characterized in that, Includes the following steps: Construct a safety-constrained manifold representation model, including collecting user information and physiological parameters, defining a set of constraint functions based on the physiological model, and defining the safety-constrained manifold as a subset of the state-action space that satisfies the constraint conditions; Based on the safety-constrained manifold, a Riemannian metric model is constructed, a set of multi-objective value functions is defined, and a Riemannian metric tensor that integrates multi-objective gradients and safety-constraint information is built to realize a dynamic weight adjustment mechanism. Using the constructed Riemannian metric model, a policy optimization algorithm based on Riemannian metric is implemented. A natural policy gradient method on the manifold is constructed, and the safety constraints are ensured through policy parameter updates and manifold constraint projection. Based on the optimized policy parameters, an adaptive Pareto optimal point switching mechanism is implemented, a Pareto front representation model is constructed, the optimal point is selected based on the sleep state, and a smooth transition mechanism is used to ensure parameter changes.

2. The pillow adaptive adjustment method based on reinforcement learning according to claim 1, characterized in that, The steps for constructing the security-constrained manifold representation model include: Collect user information and physiological parameters; The set of constraint functions is defined based on the physiological model, including cervical spine pressure constraint function, head support angle constraint function, and neck muscle tension constraint function. Define a safety-constrained manifold as a subset of the state-action space that satisfies the constraints. Implement a manifold representation algorithm to determine in real time whether a state-action pair is within a safe manifold.

3. The pillow adaptive adjustment method based on reinforcement learning according to claim 1, characterized in that, The steps for constructing the Riemann metric model include: Define a set of multi-objective value functions, including comfort value functions, health support value functions, and sleep quality value functions; Construct a Riemannian metric tensor that integrates multi-objective gradient and security constraint information; Implement a dynamic weight adjustment mechanism to adjust the weights of different objectives based on the user's sleep stage and feedback; Calculate the inverse matrix of the Riemann metric, which is used for policy optimization calculations.

4. The pillow adaptive adjustment method based on reinforcement learning according to claim 3, characterized in that, The construction of Riemannian metric tensors includes the following steps: Calculate the gradient of the value function for each optimization objective; Calculate the gradient for each constraint function; Based on the target weight and constraint weight parameters, the target gradient and constraint gradient are weighted and combined to construct the Riemann metric tensor.

5. The pillow adaptive adjustment method based on reinforcement learning according to claim 1, characterized in that, The steps for implementing the policy optimization algorithm based on Riemann metric include: Constructing natural policy gradient methods on manifolds; Implement the strategy parameter update algorithm; Safety constraints are ensured to be met through policy parameter updates and manifold constraint projection. To implement a reinforcement learning training process based on experience playback.

6. The pillow adaptive adjustment method based on reinforcement learning according to claim 5, characterized in that, The strategy parameter update process includes: Calculate the gradient of the policy loss function; The gradient is transformed using the inverse matrix of the Riemann metric; The parameters are updated based on the learning rate.

7. The pillow adaptive adjustment method based on reinforcement learning according to claim 5, characterized in that, Manifold constraint projection is achieved through the following steps: Calculate the distance between the updated parameters and the safe manifold; An iterative projection method is used to project the parameters onto the nearest safe manifold point.

8. The pillow adaptive adjustment method based on reinforcement learning according to claim 1, characterized in that, The steps for implementing the adaptive Pareto optimal point switching mechanism include: Construct a Pareto front representation model; Select the optimal point based on sleep state; A smooth transition mechanism ensures that parameter changes are handled correctly. Implement a Pareto front adaptive update algorithm based on user feedback.

9. A pillow adaptive adjustment method based on reinforcement learning according to claim 8, characterized in that, The smooth transition mechanism is achieved through the following steps: The smoothing coefficient is dynamically adjusted based on the current sleep state and depth. Based on the policy parameters from the previous time step and the policy parameters corresponding to the target Pareto optimal point, linear interpolation is performed using a smoothing coefficient to obtain the actual policy parameters to be executed.

10. A reinforcement learning-based adaptive pillow adjustment system, used to execute the reinforcement learning-based adaptive pillow adjustment method according to any one of claims 1-9, characterized in that, include: The data acquisition module, including the smart pillow actuator, sensor array system, edge computing processing unit, and communication interface module, is used to collect user sleep data and physiological parameters. The data processing module includes functions for sensor data preprocessing, state feature extraction, sleep stage identification, and physiological indicator evaluation, which are used to analyze and process the collected data. The reinforcement learning module includes constraint function set, Riemann metric construction, multi-objective weight adjustment, manifold boundary detection, policy optimization, and constraint projection functions, which are used to achieve multi-objective optimization under safe constraints. The control execution module includes functions such as user interface interaction, adjustment command execution, user feedback collection, and sleep report generation, which are used to realize the human-computer interaction and control execution of the system.

Images