Redirection walking resetting method and device based on reinforcement learning

Through VR redirection walking technology based on reinforcement learning, the deep reinforcement learning algorithm is used to optimize the reset direction and set buffers, which solves the problem of frequent users' collisions in complex physical spaces, and improves obstacle avoidance and user experience.

CN119002686APending Publication Date: 2024-11-22ZHEJIANG UNIV OF TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410567567.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing VR redirectional walking technology is difficult to effectively avoid the collision between users and obstacles in complex physical spaces, resulting in poor user experience, especially the problem of frequent collisions in narrow areas.

Method used

Using the redirected walking reset method based on reinforcement learning, the physical spatial layout is obtained through the lidar sensor, the obstacle grid map is generated, and the reset direction is optimized using the deep reinforcement learning algorithm (PPO network) to set the reset buffer, and in advance Reset the user orientation to avoid local collisions.

Benefits of technology

It significantly improves obstacle avoidance effect in complex physical spaces, allowing users to walk longer distances between two resets, improve user experience, and is suitable for various physical space layouts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119002686A_ABST
    Figure CN119002686A_ABST
Patent Text Reader

Abstract

The invention discloses a redirection walking resetting method and device based on reinforcement learning, and the method comprises the following steps: 1, obtaining a physical space layout through a laser radar sensor, generating a corresponding barrier grid map, and enabling the size of each grid to be 0.1 m * 0.1 m; for adjacent obstacles, connection processing is performed by using a corrosion and expansion technology so as to improve the structural property of the obstacle grid map; 2, simulating free roaming of the user agent in the virtual space; constructing a deep reinforcement learning network structure based on the PPO algorithm, wherein the network structure comprises an Actor network and a Critic network which are used for environment interaction, experience collection and strategy updating; design and construction of a deep reinforcement learning intelligent agent system are used for realizing intelligent decision making, environment interaction and optimization objective achievement; 5, training the neural network by using a deep reinforcement learning method; the agent interacts with the environment, collects empirical data, and inputs the empirical data into the deep neural network for training; weight parameters of the neural network are updated by optimizing the objective function, so that the performance of the agent is improved; 6, deploying the trained deep neural network into a virtual reality roaming scene; when the model judges that the user needs to be reset, roaming of the user is stopped, and the user is indicated to turn to the most appropriate direction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of virtual reality redirection, and in particular relates to a method and device for redirecting walking and resetting based on reinforcement learning. Background Art

[0002] The development of virtual reality (VR) technology has attracted great interest and applications in many fields. In traditional VR systems, users wear devices such as head-mounted displays (HMDs) and handles to interact with virtual environments. One of the important technical challenges is to enable users to walk in virtual environments while maintaining safety and comfort in the real world. Traditional VR walking methods usually use handheld controllers or somatosensory devices, but these methods may cause adverse effects such as motion sickness or motion sickness. At the same time, natural walking methods can enhance the user's experience. However, the use of real walking motion technology limits the size of VE to the tracking space. The virtual space experienced by users is infinite, while the physical tracking space is limited. This spatial difference may increase the risk of physical collisions in VR user experience. Subtle redirected walking (RDW) is a common motion technique that enables participants to truly walk in a VE larger than the tracking space by imperceptibly changing the virtual space (VE) around the user. Although subtle RDW methods try to minimize collisions imperceptibly, it is still impossible to avoid all collisions with boundaries or obstacles. Therefore, when a collision is about to occur, obvious RDW techniques must be applied to reset the user's physical direction or position. Reset technology will pause the virtual experience and guide the user to a favorable physical direction or position.

[0003] As VR gradually enters every home, users can experience VR technology in their living room or anywhere. Therefore, redirected walking technology needs to adapt to various complex physical spaces to ensure the user experience of all users. The current reset algorithms are mainly divided into two categories. One type of reset algorithm focuses more on maintaining an immersive experience rather than reducing collisions. The other type is to select the corresponding reset position and direction according to the physical space in which it is located to avoid future collisions. These reset algorithms are proposed to target rules and simple physical space layouts, which can effectively improve the obstacle avoidance effect, but still cannot guarantee the user's immersive experience, and users frequently collide with obstacles. The current reset technology still does not work very well for complex physical spaces. It is easy for users to enter narrow areas and cause frequent collisions with surrounding obstacles. Summary of the invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a method and device for resetting redirected walking based on reinforcement learning.

[0005] When users roam freely in the virtual space, collisions with physical boundaries are inevitable, so a reset strategy is needed. The algorithm hopes to help users reset to an ideal direction when they reach the boundary, allowing them to walk a relatively long distance before the next collision. At the same time, for complex environments, users can easily walk into narrow spaces and cause multiple local collisions. Therefore, by monitoring the user's position, setting a reset buffer to determine whether an early reset is needed and using an early reset strategy, a reset occurs in advance before the user enters a narrow space, turning the user to the most appropriate direction.

[0006] This is a process of generating the optimal reset direction based on the user's location and the layout of the physical space.

[0007] The first aspect of the present invention relates to a redirection walking reset method based on reinforcement learning, comprising the following steps:

[0008] 1. Use the LiDAR sensor to obtain the physical space layout and generate the corresponding obstacle grid map, with each grid size of 0.1m × 0.1m. For adjacent obstacles, use corrosion and expansion technology to connect them to improve the structure of the obstacle grid map.

[0009] 2. Simulate the free roaming of user agents in virtual space.

[0010] 3. Build a network structure for deep reinforcement learning based on the PPO algorithm, including two networks, Actor and Critic, for environment interaction, experience collection, and strategy updating.

[0011] 4. Design and construction of deep reinforcement learning intelligent agent system to achieve intelligent decision-making, environmental interaction and optimization goals. It includes the following sub-steps:

[0012] 4.1 Define Agent Control Agent: Use deep reinforcement learning methods to allow the agent to interact with the environment, collect experience, and store it in the experience replayer.

[0013] 4.2 Define the state space S: Encode the physical space as an instantaneous state vector matrix, representing the user position and obstacle layout.

[0014] 4.3 Define the action space A: including whether to reset in advance and the reset direction.

[0015] 4.4 Define the reward function R: It is used to evaluate the quality of the agent's behavior.

[0016] 5. Use deep reinforcement learning methods to train the neural network. The agent interacts with the environment, collects experience data, and inputs it into the deep neural network for training. By optimizing the objective function, the weight parameters of the neural network are updated to improve the performance of the agent.

[0017] 6. Deploy the trained deep neural network into the virtual reality roaming scene. When the model determines that the user needs to reset, stop the user's roaming and instruct the user to turn to the most appropriate direction.

[0018] The present invention proposes a redirected walking reset method based on reinforcement learning, which can not only improve the obstacle avoidance effect of simple physical space layout, but also be adapted to various complex physical spaces.

[0019] The second aspect of the present invention relates to a redirection walking reset device based on reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the redirection walking reset method based on reinforcement learning of the present invention.

[0020] The third aspect of the present invention relates to a computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, the reinforcement learning-based redirected walking reset method of the present invention is implemented.

[0021] In order to solve the reset problem of users under various physical space layouts, the present invention proposes an innovative redirection reset method based on reinforcement learning. The present invention can select the most appropriate reset direction when the user resets in all different physical space environments. The present invention can collect the physical space layout under any physical space layout and pre-process the physical space layout through corrosion and expansion. Secondly, a buffer zone around the boundary and obstacles is set to determine whether the user's current direction needs to be reset in advance under the corresponding physical space layout. Finally, if a reset is required, the user's roaming experience is stopped and the user is guided to turn to the best direction. For more complex physical spaces, early reset can be performed to avoid multiple local collisions in the area. Secondly, through corrosion and expansion graphics technology, obstacles that are close to each other are integrated, so that the method can serve complex spaces such as living rooms with multiple obstacles. This method can effectively improve the obstacle avoidance effect and thus enhance the user's experience when roaming in virtual space.

[0022] The present invention uses a reinforcement learning model to select the most appropriate direction for the current physical space when the user is reset. At the same time, for complex physical spaces, the user can be reset in advance by resetting the buffer zone to avoid multiple collisions in local areas. This method can not only improve the obstacle avoidance effect, allowing users to walk a longer distance between two collisions, but also adapt to all current redirection walking algorithms and various physical space layouts.

[0023] The advantages of the present invention are: it is applicable to any simple or complex physical space layout, and at the same time effectively improves the obstacle avoidance effect, increases the walking distance between two resets, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of erosion and expansion.

[0025] Figure 2 This is a schematic diagram of early reset.

[0026] Figure 3 It is the network training flowchart of the reinforcement learning algorithm PPO.

[0027] Figure 4 is a flow chart of the method of the present invention.

[0028] Figure 5 This is a schematic diagram of early reset. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0031] Example 1

[0032] This embodiment relates to a redirection and resetting method based on reinforcement learning, comprising the following steps:

[0033] 1. Use the LiDAR sensor to obtain the physical space layout and generate the corresponding obstacle grid map, with each grid size of 0.1m × 0.1m. For adjacent obstacles, use corrosion and expansion technology to connect them to improve the structure of the obstacle grid map.

[0034] Obstacles in physical space are diverse and their distribution is uncertain, so we first use erosion and dilation to convolve the structure element of the obstacle grid map to improve its connectivity. The step distance is set to 0.5m, and the corresponding number of grids is 5×5, so a 5×5 square kernel is used for erosion and dilation operations. Erosion can reduce the obstacle boundary and eliminate small gaps, while dilation can expand the boundary and fill the gaps. Through repeated erosion and dilation operations, adjacent obstacles can be effectively connected. Through multiple experiments and observations of image effects, the appropriate number of iterations was determined to be 5 to maintain the suitability of the processing effect. Figure 1 As shown, adjacent obstacles are connected by corrosion and expansion to facilitate subsequent processing.

[0035]

[0036] Where I is the obstacle grid map, B is the structural element (kernel), Indicates the erosion operation on the obstacle grid map.

[0037] (I⊕B)(x,y)=max (i,j)∈B I(x+i,y+j) (2) Among them, I is the obstacle grid map, B is the structural element (kernel), and (I⊕B) represents the expansion operation on the obstacle grid map.

[0038] 2. Simulate the free roaming of user agents in virtual space.

[0039] The virtual environment is mainly divided into building buildings, small exploration environments, large exploration environments, and long straight driving environments. The reinforcement learning model is trained by simulating the user's agency in these four environments.

[0040] First, the user agent is placed at a random position in the obstacle grid map, a task is designed with walking distance as the evaluation criterion, and the user agent is allowed to simulate the free roaming of humans before a certain number of collisions.

[0041] Since the average walking distance of humans is 0.5m, the walking distance of the user agent is set to 0.5m in the process of simulated roaming. For different roaming scenarios, users will have different walking trajectories, so different average straight distances and rotation angles are set in the corresponding virtual environment. The specific values ​​are shown in Table 1:

[0042] Table 1

[0043] Virtual Scene Average straight distance (meters) Steering angle (degrees) Building (2,8) (-π / 2,π / 2) Small Exploration Environment (2,6) (-π,π) Large Exploration Environment (8,12) (-π,π) Long straight environment (1000) (-π / 2,π / 2)

[0044] Set the boundaries of the obstacle grid map and the range of 0.5m around the obstacle as the collision reset zone. When the agent user enters the collision reset zone, the agent user's actions must be stopped immediately and turned to the angle generated by the model. Set the boundaries of the obstacle grid map and the range of 0.5m-1.5m around the obstacle as the reset buffer. Figure 2 As shown in the figure, the black area is the obstacle in the physical space, the red area is the collision reset area, and the yellow area is the reset buffer area. When the agent user enters the reset buffer area, the reinforcement learning model determines whether it needs to be reset in advance. If necessary, the agent user's action will be stopped immediately and turned to the angle generated by the model. Otherwise, it will continue to roam freely forward.

[0045] 3. Build a network structure for deep reinforcement learning based on the PPO algorithm, including two networks, Actor and Critic, for environment interaction, experience collection, and strategy updating.

[0046] Actor can fully explore the current environment and slowly update the strategy, while Critic only needs to evaluate the quality of the strategy. Figure 3 shown.

[0047] The input of the Actor network is the state, and the output is the action probability distribution parameter π(a t ∣s t ). An MLP (Multi-layer Perceptron) extractor is used, whose input is state features and output is potential action probabilities. The MLP extractor consists of one or more fully connected layers, and each fully connected layer is connected using the activation function Tanh. The output layer of the policy network is based on the selected action distribution type.

[0048] The Critic network contains a value function network, which consists of a fully connected layer. The input of the fully connected layer is the state feature, and the output is the estimate of the state value. The Critic network is used to evaluate the value of the state in order to calculate the advantage estimate when the policy is updated.

[0049] 4. Design and construction of deep reinforcement learning intelligent agent systems for intelligent decision-making, environmental interaction and optimization goal achievement.

[0050] 4.1 Define the redirection user agent control agent.

[0051] The user agent controls the agent using a deep reinforcement learning method to generate an agent that allows the user agent to interact with the environment and collect historical experience of the interaction. At the same time, these experiences are stored in the experience replayer in the form of (s, a, r, s'), where s represents the user's location and the layout of physical space obstacles, a represents an action selected arbitrarily from the action space A under the current physical space state s, r represents the penalty value and reward value obtained by performing action a in the current physical space state s, and s' represents the next physical space state to be migrated to after performing action a in the physical space state s.

[0052] 4.2 Define the state space S.

[0053] The physical space consists of obstacles and users, so the instantaneous state of the entire space at time t is encoded by a 1×40×40 vector matrix. The entire space is divided into 40×40 grids, the grids with obstacles are set to -1, and the grids where the users are located are set to 1. At the same time, in order to indicate the location of the user, the grids within 1m of the user are set to values ​​0-1. The farther the distance, the closer the value is to 0, and the closer the distance, the closer the value is to 1. The rest of the grids are set to 0.

[0054] The final formula is shown below.

[0055]

[0056]

[0057] Among them, Mi,j Indicates the value set for each grid, i represents the row coordinate of the grid, and j represents the column coordinate of the grid. dist((i,j), user position) represents the Euclidean distance from grid (i,j) to the user position.

[0058] In order to enable the network to understand the user's previous state and predict the future path to ensure smooth action, each observation data input to the network is a vector formed by connecting the user's 5 most recent states. The ot observation expression ot at time t is defined as: ot = [st-4, st-3, st-2, st-1, st]. When the user collides and needs to be reset, the user's most recent 5 states are equal to indicate that the user is currently stationary and can be reset at any angle.

[0059] 4.3 Define the action space A.

[0060] Each agent selects the corresponding operation according to the current state St in each time step and receives a new state St+1. In order to avoid falling into local multiple collisions, the method of early reset is adopted. Each user judges whether to perform early reset when walking to increase the walking distance of the whole process. If early reset is required or a collision has occurred, reset to the corresponding direction. The action set includes two values ​​A and B. A indicates whether early reset is required. A={0,1}, where 0 indicates that early reset is not required, and the agent continues to walk according to the corresponding redirection algorithm. 1 indicates that early reset is required. The agent will reset immediately after receiving the prompt and reset to the direction pointed by value B. B indicates the direction facing the agent after being reset when reset is required or collision has occurred. In order to reduce training time and improve the accuracy of reset direction, B is set to {0, 1..., 360}, that is, 360 degrees are divided into 360 directions, and one value represents 1°. Therefore, the final action set is Action={A,B}.

[0061] 4.4 Define the reward function R.

[0062] The reward function R includes the reset penalty function R p , New direction exploration reward function R w And the space exploration reward function R s .

[0063] For the penalty function R after reset p , the user agent cannot enter the obstacle or go outside the boundary, so when a reset occurs, the user cannot be redirected to the direction that will lead to the obstacle or the boundary in the next step. For this situation, a penalty value R is assigned p =-20.

[0064] For exploring new directions, the reward function R w In order to allow the user agent to walk the longest distance between two resets, the user agent is given a reward value of step size for each step where d i is the walking distance per step, K is a coefficient that matches the distance unit and is used to convert the distance into the unit of reward value, and N is the number of steps from reset to the next reset.

[0065] For the space exploration reward function R s , users are free to roam in the virtual space, so in order to guide users to the most idle areas to adapt to their free walking, the entire physical space is divided into 40×40 discrete cells, and the corresponding idleness is calculated for each discrete cell. The user agent gives the idleness of the current location as a reward value for each step. The idleness of each grid is defined as the following formula:

[0066]

[0067] f(x,y) represents the degree of vacancy of the discrete cell in the xth row and yth column in the physical space. If the cell contains an obstacle, the corresponding position of the matrix is ​​set to -1. Otherwise, it is proportional to the distance to the obstacle. The farther away from the obstacle, the larger the value, and the closer to the obstacle, the smaller the value.

[0068] R s =∑ k∈S f(x,y) (6)

[0069] Among them, S is the set of grids passed in one step, and F(x, y) represents the idleness of the discrete cell in the xth row and yth column.

[0070] In summary, the reward function

[0071] 5. Use deep reinforcement learning methods to train neural networks.

[0072] In each training cycle, the user agent interacts with the simulated environment, performs actions and observes the feedback (reward signals and new states) of the environment, which are stored in the experience replay memory to obtain a training sample set. These experience data are used to train the neural network.

[0073] Randomly sample the quadruple information (s, a, r, s') in the experience replay memory and input these sampled data into the deep neural network, which includes a policy network and a value network. Through the collected experience data, the advantage estimate A of each action can be calculated t =Q t -V tThe advantage estimate is the difference between the reward of the current action (including future rewards) and the value estimate of the current state.

[0074] Using advantage estimates and existing empirical data, by maximizing expected returns To update the strategy. Among them, clip(r t (θ), 1-ε, 1+ε) means pruning the strategy ratio rt(θ) to [1-ε, 1+ε]. Pruning prevents the new strategy from changing too much, thus stabilizing the algorithm.

[0075] Finally, the gradient descent algorithm Adam is used to minimize the objective function, thereby updating the weight parameters of the neural network. After multiple iterations of training, a trained deep recurrent neural network can be obtained.

[0076] 6. Deploy the trained deep neural network into the virtual reality roaming scene.

[0077] Use Unity3D and C# technology to build multiple virtual scenes, such as office buildings, communities, schools, and highways, which can simulate various environments and scenarios in the real world. Use SteamVR plug-ins to implement virtual reality (VR) functions, allowing users to experience these scenes immersively through head-mounted displays and handheld controllers. Through virtual reality technology, users can experience and train in a safe environment, such as learning, training, and entertainment.

[0078] After the deep neural network training is completed, the model is deployed in the virtual reality roaming scene, allowing users to enter the virtual environment wearing a head-mounted display and roam freely. The user's position is collected in real time through the locator and the user's walking path is input into the model in real time. When the model determines that the user needs to be reset, the user's roaming is immediately stopped and the user is instructed to turn to the direction predicted by the model. The entire flow chart is as follows Figure 4 shown.

[0079] Example 2

[0080] The present embodiment relates to a redirection walking reset device based on reinforcement learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the redirection walking reset method based on reinforcement learning of Example 1.

[0081] Example 3

[0082] The present embodiment relates to a computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, the redirection walking reset method based on reinforcement learning of Example 1 is implemented.

[0083] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A redirected walking reset method based on reinforcement learning includes the following steps: S1. Use the LiDAR sensor to obtain the physical space layout and generate the corresponding obstacle grid map. The size of each grid is 0.1m×0.1m. For adjacent obstacles, use corrosion and expansion technology to connect them to improve the structure of the obstacle grid map. S2. Simulate the free roaming of user agents in the virtual space; S3. Build a network structure for deep reinforcement learning based on the PPO algorithm, including two networks, Actor and Critic, for environment interaction, experience collection, and strategy update; S4. Design and construction of a deep reinforcement learning intelligent agent system to achieve intelligent decision-making, environmental interaction and optimization goals; specifically includes the following sub-steps: 4.1 Define agent control agent: Use deep reinforcement learning methods to let the agent interact with the environment, collect experience, and store it in the experience replayer; 4.2 Define the state space S: Encode the physical space into an instantaneous state vector matrix, representing the user position and obstacle layout; 4.3 Define the action space A: including whether to reset in advance and the reset direction; 4.4 Define the reward function R: used to evaluate the quality of agent behavior; S5. Use deep reinforcement learning methods to train neural networks; The agent interacts with the environment, collects experience data, and inputs it into the deep neural network for training; by optimizing the objective function, the weight parameters of the neural network are updated to improve the performance of the agent; S6. deploy the trained deep neural network into the virtual reality roaming scene; When the model determines that the user needs to reset, it stops the user's roaming and instructs the user to turn to the most appropriate direction.

2. The method for redirecting walking and resetting based on reinforcement learning as claimed in claim 1, Features: Step S1 specifically includes: The step distance is set to 0.5m, and a 5×5 square kernel is used for erosion and dilation operations; adjacent obstacles are connected through repeated erosion and dilation operations; the number of iterations is determined to be 5 times to maintain the appropriateness of the processing effect, and adjacent obstacles are connected through erosion and dilation; Where I is the obstacle grid map, B is the structural element (kernel), Indicates the erosion operation on the obstacle grid map; Where I is the obstacle grid map, B is the structural element (kernel), Indicates the expansion operation on the obstacle grid map.

3. The method for redirecting walking and resetting based on reinforcement learning as claimed in claim 1, Features: Step S2 specifically includes: The virtual environment is divided into building building, small exploration environment, large exploration environment and long straight environment. The reinforcement learning model is trained by simulating the user's agency in these four environments; First, the user agent is placed at a random position in the obstacle grid map, and a task is designed with walking distance as the evaluation criterion, and the user agent is allowed to simulate the free roaming of humans before a certain number of collisions. Since the average walking distance of humans is 0.5m, the walking distance of the user agent is set to 0.5m in the process of simulated roaming. For different roaming scenarios, users have different walking trajectories, so different average straight distances and rotation angles are set in the corresponding virtual environment. The four boundaries of the obstacle grid map and the range of 0.5m around the obstacle are set as the collision reset zone. When the agent user enters the collision reset zone, the agent user's action needs to be stopped immediately and the agent user needs to turn to the angle generated by the model; the four boundaries of the obstacle grid map and the range of 0.5m-1.5m around the obstacle are set as the reset buffer zone; as shown in Figure 2, the black area is the obstacle in the physical space, the red area is the collision reset zone, and the yellow area is the reset buffer zone; when the agent user enters the reset buffer zone, the reinforcement learning model is used to determine whether a reset is required in advance. If necessary, the agent user's action needs to be stopped immediately and the agent user needs to turn to the angle generated by the model. Otherwise, the agent user continues to roam freely forward.

4. The method for redirecting walking and resetting based on reinforcement learning as claimed in claim 1, Features: Step S3 specifically includes: The Actor fully explores the current environment and slowly updates its strategy, while the Critic is responsible for evaluating the quality of the strategy; The input of the Actor network is the state, and the output is the action probability distribution parameter π(a t ∣s t ); an MLP (multi-layer perceptron) extractor is used, whose input is state features and output is potential action probabilities; the MLP extractor consists of one or more fully connected layers, each of which is connected using the activation function Tanh; the output layer of the policy network is based on the selected action distribution type The Critic network contains a value function network consisting of a fully connected layer; the input of the fully connected layer is the state feature, and the output is an estimate of the state value; the Critic network is used to evaluate the value of the state in order to calculate the advantage estimate when the strategy is updated.

5. The method for redirecting walking and resetting based on reinforcement learning as claimed in claim 1, Features: Step S4 specifically includes: S41. Define redirection user agent control Agent; The user agent control agent uses a deep reinforcement learning method to generate an agent, allowing the user agent to interact with the environment and collect historical experience of the interaction, and store these experiences in the experience replay device in the form of (s, a, r, s'), where s represents the user's location and the layout of physical space obstacles, a represents an action selected arbitrarily from the action space A under the current physical space state s, r represents the penalty value and reward value obtained by performing action a in the current physical space state s, and s' represents the next physical space state to migrate to after performing action a in the physical space state s; S42. Define the state space S; The physical space is composed of obstacles and users, so the instantaneous state of the entire space at time t is encoded by a 1×40×40 vector matrix; the entire space is divided into 40×40 grids, the grids with obstacles are set to -1, and the grids where the users are located are set to 1; at the same time, in order to indicate the location of the user, the grids within 1m of the user are set to values ​​0-1, the farther the distance, the closer the value is to 0, the closer the distance, the closer the value is to 1, and the rest of the grids are set to 0; The final formula is as follows; Among them, M i,j Indicates the value set for each grid, i represents the row coordinate of the grid, and j represents the column coordinate of the grid; dist((i,j), user position) represents the Euclidean distance from the grid (i,j) to the user position; Each observation data input to the network is a vector formed by connecting the user's five most recent states; the ot observation expression ot at time t is defined as: ot = [st-4, st-3, st-2, st-1, st]; when the user collides and needs to be reset, the user's five most recent states are equal to indicate that the user is currently stationary and can be reset at any angle; S43. Define action space A; Each agent selects the corresponding operation according to the current state St in each time step, and receives a new state St+1; in order to avoid falling into local multiple collisions, the method of early reset is adopted. Each user judges whether to perform early reset when walking to increase the walking distance of the whole process. If early reset is required or a collision has occurred, reset to the corresponding direction; the action set includes two values ​​A and B, A indicates whether early reset is required A={0,1}, where 0 indicates that early reset is not required, and the agent continues to walk according to the corresponding redirection algorithm, and 1 indicates that early reset is required. After receiving the prompt, the agent will reset immediately and reset to the direction pointed to by value B; B indicates the direction facing the agent after being reset when reset is required or collision has occurred; in order to reduce training time and improve the accuracy of reset direction, B is set to {0,1...,360}, that is, 360 degrees are divided into 360 directions, and one value represents 1°; therefore, the final action set is Action={A,B}; S44. Define a reward function R; The reward function R includes the reset penalty function R p , New direction exploration reward function R w And the space exploration reward function R s ; For the penalty function R after reset p , the user agent cannot enter the obstacle or go outside the boundary, so when a reset occurs, the user cannot be redirected to the direction that will lead to the obstacle or the boundary in the next step. For this situation, a penalty value R is assigned p =-20; For exploring new directions, the reward function R w , in order to allow the user agent to walk the longest distance between two resets, the user agent is given a reward value of step size for each step where d i is the walking distance of each step, K is a coefficient that matches the distance unit and is used to convert the distance into the unit of reward value, and N is the number of steps from reset to the next reset; For the space exploration reward function R s , users are free to roam in the virtual space, so we need to guide users to the most idle areas to adapt to their free walking. The entire physical space is divided into 40×40 discrete cells, and the corresponding idleness is calculated for each discrete cell. The user agent gives the idleness of the current location as a reward value for each step he takes. The idleness of each grid is defined as the following formula: f(x,y) represents the degree of vacancy of the discrete cell in the xth row and yth column in the physical space; if the cell contains an obstacle, the corresponding position of the matrix is ​​set to -1, otherwise it is proportional to the distance to the obstacle. The farther away from the obstacle, the larger the value, and the closer to the obstacle, the smaller the value; Rs=∑kesf(x,y)(6) Among them, S is the set of grids passed in one step, and F(x, y) represents the idleness of the discrete cell in the xth row and yth column; 6. The method for redirecting walking and resetting based on reinforcement learning as claimed in claim 1, Features: Step S5 specifically includes: Use Unity3D and C# technologies to build multiple virtual scenes to simulate various environments and scenarios in the real world; use SteamVR plug-ins to implement virtual reality (VR) functions, allowing users to experience these scenes immersively through head-mounted displays and handheld controllers; through VR technology, users can conduct various experiences and training in a safe environment; After the deep neural network training is completed, the model is deployed in a virtual reality roaming scene, allowing the user to enter the virtual environment wearing a head-mounted display and roam freely; the user's position is collected in real time through the locator and the user's walking path is input into the model in real time. When the model determines that the user needs to be reset, the user's roaming is immediately stopped and the user is instructed to turn to the direction predicted by the model.

7. A redirection walking reset device based on reinforcement learning, It is characterized in that It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the redirection walking reset method based on reinforcement learning as described in any one of claims 1-6.

8. A computer-readable storage medium, It is characterized in that A program is stored thereon, and when the program is executed by a processor, the redirected walking reset method based on reinforcement learning described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • VR redirection walking method and system based on deep reinforcement learning

    CN122223281A