A method for controlling a bed cover based on deep reinforcement learning

Abstract

The application provides a deep reinforcement learning-based convenient shield control method, which comprises the following steps: S1, modeling of a working process of automatic adhesion of a non-wearable convenient shield to a human body in deep reinforcement learning; S2, obtaining a training target of deep reinforcement learning through definition of deep reinforcement learning framework elements in S1; S3, guiding the non-wearable convenient shield to reach a target state based on a reward function of a pressure combination state; and S4, optimizing network parameters through gradient updating to achieve the purpose of automatic adhesion of the non-wearable convenient shield to the human body. In view of the problems of inaccurate modeling and poor self-adaptability of a continuous body shield using a traditional control method, the deep reinforcement learning algorithm is applied to the control process of the continuous body shield, the complex modeling and control algorithm derivation process is omitted, a reinforcement learning control framework is built, a black box model is established, a strategy network of an intelligent agent is trained, the output strategy of the intelligent agent is gradually optimized, and the purpose of improving the self-adaptability of the shield is achieved.

Description

Technical Field

[0001] This application belongs to the field of next-generation information technology and relates to an automatic flushing control method for nursing beds, specifically a method for controlling the toilet flushing cover based on deep reinforcement learning. Background Technology

[0002] With the accelerating aging of my country's population, the number of bedridden and disabled elderly people is rapidly increasing. Most bedridden and disabled elderly people also suffer from bowel and bladder dysfunction, which places higher demands on other family members and social elderly care service institutions. Bowel and bladder care robots are a type of care robot developed for elderly people with limited mobility, disabilities, or partial disabilities. They can automatically detect the patient's urine and feces and perform operations such as flushing, washing, and drying, helping the elderly achieve self-care in terms of urination and defecation. This alleviates the embarrassment and unpleasant, tiring, and unsanitary working environment faced by caregivers when dealing with the elderly's bowel and bladder issues. It also solves problems such as the susceptibility to infection and difficulty in cleaning during patient care, effectively replacing caregivers in bowel and bladder care.

[0003] Existing cleaning and care robots are mainly wearable devices. Because the current technology requires close contact with the body during use, the feeling of constraint is too strong, which restricts their application and promotion, and has not yet achieved the goal of reducing the burden of manual care.

[0004] Existing technology shows that non-wearable toilet cleaning robots are comfortable and unrestricted, but their protective covers are not adequate during rinsing, allowing wastewater to splash and contaminate the mattress. Therefore, non-wearable covers need to automatically adapt to the user's defecation posture, fitting closely to the body to form a sealed cavity for subsequent cleaning. If the contact force is too weak, the cover will not fit effectively; if the contact force is too strong, it will cause injury.

[0005] A search revealed a published patent application (Publication No. CN116270073A) for an automatic flushing toilet cover and its operating method for a nursing bed. This patent application includes a cover frame with multiple frame joints arranged sequentially from top to bottom. The left and right sides of the front ends of adjacent frame joints are hinged together by frame pivots, and the rear ends of adjacent frame joints are hinged together by telescopic devices. The left and right sides of the front ends of the lowest frame joint are hinged to the left and right sides of the middle of the base by frame pivots, and its rear end is hinged to the rear of the base by a telescopic device. Buffer devices are installed on the edges of each frame joint and on the edges of the base that will come into contact with the human body. This automatic flushing toilet cover can be used in automatic flushing toilets on nursing beds. It can deform and adjust according to the user's body posture, effectively preventing urine from splashing and soiling the mattress when the user urinates or defecating, or liquid from splashing and wetting the mattress when flushing. The patent application uses airbag cushioning to improve sealing and cushioning performance, which is completely different from the technical means used in this application.

[0006] Therefore, designing a defecation coordination docking control algorithm with controllable human-machine contact force to achieve seamless coordination docking between the human body and the defecation receiving cover can play an important role in promoting the research and development of non-wearable defecation cleaning and care robots. Summary of the Invention

[0007] The purpose of this application is to overcome the shortcomings of the prior art and provide a contact shield control method based on deep reinforcement learning.

[0008] A method for controlling a contact shield based on deep reinforcement learning includes the following steps:

[0009] S1 models the working process of non-wearable toilet seat covers automatically fitting the human body in deep reinforcement learning. The problem is modeled as a sequential decision problem. In order to ensure that the neural network model can perform a large number of interactions with the environment, the definition of deep reinforcement learning framework elements is set, including state space, action space, and reward.

[0010] S2, through the definition of deep reinforcement learning framework elements in S1, models the continuous decision space combinatorial optimization problem in temporal tasks as a Markov decision process and derives the training objective of deep reinforcement learning.

[0011] S3 is a reward function based on the stress combination state, which uses the reward of the training target in multiple S2 to guide the non-wearable contact shield to reach the target state.

[0012] S4 optimizes the network parameters by repeatedly performing gradient updates in step S3, gradually obtaining the optimal expected return, and finally obtaining the optimal solution to the continuous decision optimization problem, thus achieving the goal of automatic body fit for the non-wearable contact shield.

[0013] Furthermore, in S1, the current pressure combination state is used as the state space, and the pressure combination is the pressure information between the edge of the shield and the human body, and its expression is as follows:

[0014] S={S n}

[0015] Where S is the state space, S n It is a sub-element in the state space, namely the pressure information state of the edge of the shield.

[0016] Furthermore, in S1, the driving input under continuous process is discretized into the driving input under sequence space as the workspace. The driving input is the driving torque of each joint of the continuous body shield. The action space formula is:

[0017] A={A n}

[0018] Where A is the action space, An It is a sub-element in the action space, that is, the joint driving torque in the sequence state.

[0019] Furthermore, in S1, the expression for the reward is as follows:

[0020]

[0021] Among them, R t For cumulative rewards, r i γ represents the immediate reward, referring to the reward sequence obtained throughout the entire exercise round starting from time t. γ represents the discount factor, with a value range of [0, 1], indicating the importance of future rewards to the present. The larger the influence factor value, the smaller the impact of future states and actions on the current cumulative reward.

[0022] Moreover, in S2, the continuous decision space combinatorial optimization problem in the temporal task is modeled as a Markov decision process, and the training objective of deep reinforcement learning is derived.

[0023] The reinforcement learning process exhibits the Markov property, meaning that the information of the current state is a synthesis of information from all historical states. Throughout the interaction, the non-wearable contact shield acts as an agent, observing the current state S at time t. t Based on the strategy, select an action, observe the action, and obtain a new state S. t+1 Finally, the reward for the previous action is calculated based on the state, and then the next decision is made; the ultimate goal of the agent is to find the optimal strategy and obtain the maximum reward.

[0024] Furthermore, in S3, the reward function based on the combined stress state is as follows:

[0025] r(s) = r suc (s)+r f (s)+r rest (s)+r pos (s)

[0026] The reward function mainly consists of four parts, namely the success reward function r suc (s), force error reward function r f (s), range-limited reward function r rest (s) and contact location reward function r pos (s), where the success reward function limits the expected force, the force error function limits the error between the expected force and the real-time force, the range limitation reward function limits the exploration range of the agent, and the contact position reward function limits the importance of different contact positions.

[0027] Furthermore, the success reward function limits the error value between the expected force and the expected force. When the real-time contact force reaches the expected force error range, a success reward can be obtained. The larger the success reward setting, the better the force tracking performance of the agent. The success reward function is expressed as follows:

[0028]

[0029] Where s min s max These are the lower and upper bounds of the error range between the real-time contact force and the expected force.

[0030] Furthermore, the force error reward function defines the error between the expected force and the real-time force, and it comprises two parts:

[0031] The first part calculates the error between the real-time contact force and the expected contact force. Under the excitation of this function, the intelligent system continuously adjusts towards the expected contact force based on the acquired real-time contact force. The smaller the difference between the real-time contact force and the expected contact force, the greater the reward value it obtains.

[0032] The second part calculates the error between the real-time contact force and the previous contact force. During the contact process, the contact force fluctuates due to changes in position. Therefore, under the excitation of this function, the fluctuation of the contact force will continuously decrease, which can be expressed as:

[0033]

[0034] Where f now For real-time contact force, f desire For the desired contact force, f last The contact force at the previous sampling time.

[0035] Furthermore, the range-limited reward function restricts the agent's exploration range; once the set range is exceeded, the task ends, ensuring training efficiency. It is expressed as:

[0036] r rest (s)=-θ

[0037] The contact location reward system assigns different levels of importance to different pressure combinations, and provides different rewards based on the region of the accumulated pressure value, as shown below:

[0038]

[0039] Moreover, in S4, the network parameters are optimized through gradient updates, the agent is trained using a deep deterministic strategy gradient algorithm, the optimal expected return is gradually obtained, and finally the optimal solution to the continuous decision optimization problem is obtained, so as to achieve the purpose of automatic fit of the non-wearable contact shield to the human body.

[0040] In summary, this application includes at least one of the following beneficial technical effects:

[0041] 1. A control method for non-wearable contact shields that automatically conform to the human body based on deep reinforcement learning is provided. Addressing the problems of inaccurate modeling and poor adaptability of traditional control methods for continuum shields, this method applies deep reinforcement learning algorithms to the control process of the continuum shield, eliminating the complex modeling and derivation of control algorithms. A reinforcement learning control framework is built, a black-box model is established, the policy network of the agent is trained, and the agent's output policy is gradually optimized to improve the shield's adaptability.

[0042] 2. A reward function based on pressure combination states is provided. By utilizing the pressure information provided by the shield, the contact state of the shield during the process of fitting the human body is captured. The reward mechanism based on pressure combination states encourages the agent to conduct meaningful exploration. Guided by the reward mechanism based on pressure combination states, the agent will not lose effective feedback information due to random exploration, thereby improving the compliance of the shield movement. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the continuous shield structure in Example 1.

[0044] Figure 2 This is a schematic diagram of the continuous body shield in use in Example 1.

[0045] Figure 3 This is a distribution diagram of the pressure sensing device in Example 1.

[0046] Figure 4 This is the flowchart of the control algorithm.

[0047] Figure 5 This is a schematic diagram of the location of the intelligent agent.

[0048] Figure 6 This is a schematic diagram of the continuous shield structure in Example 2.

[0049] Figure 7 This is a schematic diagram of the continuous body shield in use in Example 2.

[0050] Figure 8 This is a distribution diagram of the pressure sensing device in Example 2. Detailed Implementation

[0051] The present application will be further described in detail below with reference to the accompanying drawings.

[0052] The core of control for non-wearable toilet cleaning robots lies in the control of the protective shield. During shield control, the contact force generated by the shield's contact with the human body is crucial for ensuring human-machine safety and comfort. Researching a contact force compliance control algorithm based on deep reinforcement learning to enable the toilet cleaning and care robot to automatically and closely contact the human body is of great significance for meeting the safe and unrestricted toilet cleaning and care needs of bedridden elderly, reducing the labor intensity and difficulty for caregivers, and improving the quality of life and dignity of bedridden elderly.

[0053] Example 1

[0054] The structure of the continuous body shield in this embodiment is as follows: Figure 1 As shown, the continuous shield includes a shield base and two shield frames. The left and right sides of the middle part of the shield base are respectively hinged to the two front sides of the lower end of the first shield frame through the frame pivot. The two front sides of the upper end of the first shield frame are respectively hinged to the two front sides of the lower end of the second shield frame through the frame pivot.

[0055] The frame's pivot points can be controlled to rotate via input commands, thereby controlling the bending of each of the two frame sections and ultimately the continuous protective cover, causing it to deform and conform closely to the human body. The fit to the human body is as follows: Figure 2 As shown.

[0056] like Figure 3 As shown, pressure sensors are distributed at the edges of the shield where it contacts the human body. These sensors detect the contact state between the shield and the human body and provide feedback on the magnitude of the contact force. If the contact force is too small, the shield cannot fit tightly and effectively; if the contact force is too large, it will cause damage to the body.

[0057] Distribution of pressure sensing devices as follows Figure 3 As shown, pressure sensing devices 1, 2, and 6 are used to sense the contact state between the continuous protective cover and the human body during the bending process, and to provide timely feedback on the magnitude of the contact force between the protective cover and the human body; pressure sensing devices 3, 4, and 5 are used to detect the contact state between the human buttocks and the base of the protective cover, and to detect whether the human's sitting position is located at the center line of the protective cover.

[0058] The pressure combination states are shown in the figure. Pressure sensing device 1 is combination one, and pressure sensing devices 2 and 6 are combination two.

[0059] To achieve end-to-end control of input and output, and to achieve seamless coordination between the human body and the toilet cover, a deep reinforcement learning-based control method for the toilet cover is proposed. This method optimizes the output of the continuous decision space by designing a reward function based on pressure-combined state feedback. A gradient descent optimization algorithm is used to train the agent to update the policy network, resulting in a measurement network that maximizes the reward for the optimization task. This measurement network is then applied to a non-contact continuous body cover to achieve end-to-end control, enabling the non-contact cover to automatically conform to the human body.

[0060] The specific steps are as follows:

[0061] S1 requires modeling the temporal tasks in real-world applications using deep reinforcement learning, transforming the problem into a sequential decision-making problem. Specifically, it transforms the real-world process of a non-contact continuum shield fitting onto the human body into environment modeling within reinforcement learning. Correspondingly, this real-world process of the non-contact shield automatically fitting onto the human body is defined using reinforcement learning framework elements, including state space, action space, and reward, to meet the need for multiple interactions between the agent and the environment model.

[0062] This method employs deep reinforcement learning, and, while ensuring the correctness and reasonableness of the elements corresponding to the problem, abstracts the definitions of these elements, namely the state space, action space, and reward function. For example... Figure 5 As shown, in deep reinforcement learning, the agent randomly samples an action from the action space based on its current state in the state space. It then interacts with the environment using this action. The environment evaluates the action according to a predefined reward function and returns a corresponding reward value. The agent evaluates the received reward from the environment for this action to obtain a higher reward in the next interaction, thus forming an interaction process. Simultaneously, the state is updated in the next time step, and the above operations are repeated, thereby achieving a continuous interaction process.

[0063] in:

[0064] Specifically, in step 1, the state information of deep reinforcement learning, i.e., the state space, contains the environmental information perceived by the agent, as well as the changes brought about by its own actions after interacting with the environment. State information is the basis for the agent to update its decisions and evaluate its rewards. The quality of the state design directly determines whether the deep reinforcement learning algorithm can converge, its convergence speed, and its final performance. The state space is the set containing all the designed state information. The pressure combination refers to the pressure information of the edge of the shield in contact with the human body, expressed by the following formula.

[0065] S={S n}

[0066] In the formula, S describes the agent's perception of the environment, including the characteristics of the time-series task data, i.e. the information of each pressure combination state. In this invention, S is an n-dimensional numerical vector representing the pressure of each pressure combination state.

[0067] Specifically, in step 1, the driving input under the continuous process is discretized into the driving input under the sequence space as the workspace. The action space contains the set of all actions that can be sampled in the task. The driving input is the driving torque of each joint of the continuum shield, and the action space formula is:

[0068] A={A n}

[0069] In the formula, A describes the input action of the agent to the environment, through which it interacts with the environment, that is, the driving input of the sequence state.

[0070] Specifically, in step 1, the reward function is denoted as R. t In each time series, the agent makes a decision based on its policy's state at the current moment and selects an action from the action space. The environment then responds accordingly, triggering a state transition to a new environment. Simultaneously, along with the state transition, the system generates a reward signal. This reward signal is typically presented numerically, and its calculation usually involves the application of a reward function. The reward function serves as an incentive and guide for the agent to learn a favorable policy. The environment uses the reward function to evaluate the actions derived from the policy network and generate the corresponding reward value for that action. The expression for the reward is as follows:

[0071]

[0072] Among them, R t For cumulative rewards, r i γ represents the immediate reward, referring to the reward sequence obtained throughout the entire movement round starting from time t. γ represents the discount factor, with a value range of [0, 1], indicating the importance of future rewards to the present. The larger the influence factor value, the smaller the impact of future states and actions on the current cumulative reward. Reward is the goal that the agent wants to maximize when selecting action elements to execute in the action space.

[0073] Based on the deep reinforcement learning framework established in Step 1, and the definitions of action space, state space, and reward, S2 models the continuous decision-making pressure feedback combinatorial optimization problem in a temporal task as a Markov decision process, and derives the training objective of deep reinforcement learning. The reinforcement learning process exhibits the Markov property, meaning that the information of the current state is a synthesis of information from all historical states. Throughout the interaction, the non-wearable contact shield acts as the agent, observing the current state S at time t. tThen the agent selects an action from the action space according to the policy network and observes the new state S. t+1 Finally, the reward for the previous action is calculated based on the state, and then the next decision is made. The ultimate goal of the intelligent agent is to find the optimal policy network and maximize the reward through a gradual interaction process, that is, multiple iterations of training, updating the policy network parameters one by one.

[0074] In this step, the problem is modeled as a Markov decision process. In reinforcement learning, the agent and the environment need to interact continuously. After receiving feedback from the environment, the agent selects an action from the action space through a policy network and returns the selected action to the environment. After receiving the action selected by the agent, the environment enters the next state, generates a reward, and transmits the next state back to the agent. At the beginning of the learning process, the agent has an initial state S0. Under the initialized policy, it selects an action A0, thereby changing its own state. This selection process is quantified by transition probabilities. After making a selection and executing the action, it receives a feedback R0, which is used to evaluate the quality of the agent's performance. This process continues until the maximum reward is obtained.

[0075] The reward function for S3 based on the stress combination state is as follows:

[0076] r(s) = r suc (s)+r f (s)+r rest (s)+r pos (s)

[0077] The reward function mainly consists of four parts, namely the success reward function r suc (s), force error reward function r f (s), range-limited reward function r rest (s) and contact location reward function r pos (s), where the success reward function limits the expected force, the force error function limits the error between the expected force and the real-time force, the range limitation reward function limits the exploration range of the agent, and the contact position reward function limits the importance of different contact positions.

[0078] Furthermore, the success reward function limits the error value between the expected force and the expected force. When the real-time contact force reaches within this expected force error range, a success reward is obtained. The larger the success reward setting, the better the force tracking performance of the agent. The success reward is represented as follows:

[0079]

[0080] Where s min s maxThese are the lower and upper bounds of the error range between the real-time contact force and the expected force.

[0081] Furthermore, the force error reward function defines the error between the desired force and the real-time force, and it comprises two parts. The first part calculates the error between the real-time contact force and the desired contact force. Under the excitation of this function, the intelligent system continuously adjusts towards the desired contact force based on the acquired real-time contact force. The smaller the difference between the real-time contact force and the desired contact force, the greater the reward value it acquires. The second part calculates the error between the real-time contact force and the previous contact force. During contact, the contact force fluctuates due to position changes, so under the excitation of this function, the fluctuation of the contact force will continuously decrease. This can be expressed as:

[0082]

[0083] Where f now For real-time contact force, f desire For the desired contact force, f last The contact force at the previous sampling moment.

[0084] Furthermore, the range-limited reward function restricts the agent's exploration range; once the set range is exceeded, the task ends, ensuring training efficiency. It is expressed as:

[0085] r rest (s)=-θ

[0086] Furthermore, the contact location reward sets different levels of importance for different pressure combination states, and gives different rewards based on the region of the accumulated pressure state value, as shown below:

[0087]

[0088] S4 optimizes network parameters through gradient updates, employing a deep deterministic policy gradient algorithm to train the agent, gradually deriving the optimal expected reward, and finally obtaining the optimal solution to the continuous decision optimization problem. The main purpose of this step is to train the agent's policy network, enabling it to gradually update network parameters and select the optimal parameters to achieve an optimal expected reward, thus achieving the goal of automatic body conformity of the non-wearable contact shield. Since the shield conforming to the body is a continuous motion control problem, this invention uses a deep deterministic policy gradient algorithm to train the agent.

[0089] Specifically, the Deep Deterministic Policy Gradient Algorithm (DPRQA) is an offline policy training algorithm based on the Actor-Critic framework. It uses two different networks for action selection and evaluation. The Actor network directly learns the policy, receiving the current state S from the environment and outputting the corresponding action based on the policy. The Critic network, on the other hand, calculates the Q-value based on the current state and the action to evaluate the quality of the action. During training, the Actor network explores the environment through behavioral policies, which are the target policy with added noise.

[0090] Specifically, the evaluation network updates itself by minimizing the error between the actual action-state values ​​and the estimated action-state values. This is represented as follows:

[0091] L c =r t +γQ(s t+1 ,π(s t+1 ))-Q(s t a t )

[0092] Where, r t +γQ(s t+1 ,π(s t+1 )) represents the actual action state value; Q(s) t a t ) represents the estimated action state value.

[0093] Specifically, the Actor uses policy gradient descent to train the network via gradient descent of the loss function, which is expressed as:

[0094] L a =-E s [Q(s, π(s))]

[0095] Here, s is sampled from the empirical replay pool. The gradient of the loss function with respect to the parameters of the Actor network can be calculated by backpropagation combining the Critic and Actor networks.

[0096] Example 2

[0097] like Figure 6 As shown, based on specific embodiment 1, the structure of the continuous body shield is improved by increasing the number of shield skeletons to four sections, thereby increasing the controllable degrees of freedom of the continuous body shield and increasing the bending position to better fit the human body.

[0098] A schematic diagram illustrating the process of the four-joint continuous body shield fitting the human body is shown below. Figure 7As shown, specifically, the number of pressure sensing devices at the edges of the shield where they contact the human body has also increased accordingly.

[0099] like Figure 8 As shown, pressure sensing devices 1-10 are used to sense the contact state between the continuous protective cover and the human body during the bending process, and to provide timely feedback on the magnitude of the contact force between the protective cover and the human body. Pressure sensing devices 5-7 are used to detect the contact state between the human buttocks and the base of the protective cover, and to detect whether the human's sitting position is located at the center line of the protective cover.

[0100] Specifically, pressure sensing device 1 is combination one, pressure sensing devices 2 and 10 are combination two, pressure sensing devices 3 and 9 are combination three, and pressure sensing devices 4 and 8 are combination four.

[0101] The control method and steps for the four-joint continuous body shield to fit the human body are as described in Example 1.

[0102] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for controlling a contact shield based on deep reinforcement learning, characterized in that: Includes the following steps: S1 models the working process of non-wearable colostomy shields automatically fitting the human body in deep reinforcement learning. The problem is modeled as a sequential decision problem. In order to ensure that the neural network model can perform a large number of interactions with the environment, the definition of deep reinforcement learning framework elements is set, including state space, action space, and reward. Using the current pressure combination state as the state space, the pressure combination is the pressure information between the edge of the shield and the human body in contact, and its expression is as follows: in, For state space, This refers to a sub-element in the state space, specifically the pressure information state of the contact area at the edge of the shield. S2, through the definition of deep reinforcement learning framework elements in S1, models the continuous decision space combinatorial optimization problem in temporal tasks as a Markov decision process and derives the training objective of deep reinforcement learning. S3 is a reward function based on the stress combination state, which uses the reward of the training target in S2 to guide the non-wearable contact shield to reach the target state. The reward function based on the stress combination state is as follows: The reward function mainly consists of four parts, namely the success reward function. Force error reward function Range-limited reward function and contact location reward function Among them, the success reward function limits the expected force, the force error function limits the error between the expected force and the real-time force, the range limit reward function limits the exploration range of the agent, and the contact position reward function limits the importance of different contact positions. The success reward function limits the error value between the expected force and the expected force. When the real-time contact force reaches the expected force error range, a success reward is obtained. The larger the success reward setting, the better the force tracking performance of the agent. The success reward function is expressed as follows: in , These are the lower and upper bounds of the error range between the real-time contact force and the expected force; The force error reward function defines the error between the expected force and the real-time force, and it consists of two parts: The first part calculates the error between the real-time contact force and the expected contact force. Under the excitation of this function, the intelligent system continuously adjusts towards the expected contact force based on the acquired real-time contact force. The smaller the difference between the real-time contact force and the expected contact force, the greater the reward value it obtains. The second part calculates the error between the real-time contact force and the previous contact force. During the contact process, the contact force fluctuates due to changes in position. Therefore, under the excitation of this function, the fluctuation of the contact force will continuously decrease, which can be expressed as: in For real-time contact force, For the desired contact force, The contact force at the previous sampling time; The range-limited reward function restricts the agent's exploration range. Once the set range is exceeded, the task ends, ensuring training efficiency. It is expressed as: The contact location reward system assigns different levels of importance to different pressure combinations, and provides different rewards based on the region of the accumulated pressure value, as shown below: ； S4 optimizes the network parameters by using gradient updates in step S3, gradually obtaining the optimal expected return, and finally obtaining the optimal solution to the continuous decision optimization problem, so as to achieve the purpose of automatic fit of the non-wearable contact shield to the human body.

2. The contact shield control method based on deep reinforcement learning according to claim 1, characterized in that: In S1, the driving input under the continuous process is discretized into the driving input under the sequence space as the workspace. The driving input is the driving torque of each joint of the continuous body shield. The action space formula is: in, For the action space, It is a sub-element in the action space, that is, the joint driving torque in the sequence state.

3. The connection shield control method based on deep reinforcement learning according to claim 1, characterized in that: In S1, the expression for the reward is as follows: in, To accumulate rewards, γ represents the immediate reward, referring to the reward sequence obtained throughout the entire exercise round starting from time t. γ represents the discount factor, with a value range of [0,1], indicating the importance of future rewards to the present. The larger the influence factor value, the smaller the impact of future states and actions on the current cumulative reward.

4. The contact shield control method based on deep reinforcement learning according to claim 1, characterized in that: In S2, the continuous decision space combinatorial optimization problem in the time-series task is modeled as a Markov decision process, and the training objective of deep reinforcement learning is derived. The reinforcement learning process exhibits the Markov property, meaning that the information of the current state is a synthesis of the information of all historical states. Throughout the interaction, the non-wearable contact shield acts as an intelligent agent. t Observe the current state at all times. Choose an action based on the strategy, observe the action, and obtain a new state. Finally, the reward for the previous action is calculated based on the state, and then the next decision is made; the ultimate goal of the agent is to find the optimal strategy and obtain the maximum reward.

5. The contact shield control method based on deep reinforcement learning according to claim 1, characterized in that: In step S4, network parameters are optimized through gradient updates, and the agent is trained using a deep deterministic strategy gradient algorithm to gradually obtain the optimal expected return. Finally, the optimal solution to the continuous decision optimization problem is obtained, achieving the goal of automatic fit of the non-wearable contact shield to the human body.

Images