Indoor navigation method and device, computer device, storage medium and program product

Abstract

The application relates to an indoor navigation method and device, a computer device, a storage medium and a program product. The method comprises the following steps: acquiring environment data and positioning data of an autonomous mobile device in a real environment; the environment data comprises a depth image collected by a depth sensor of the autonomous mobile device; the positioning data comprises position data of the autonomous mobile device, a heading angle and position data of a target point; the environment data and the positioning data are input into a pre-trained navigation model to generate an action control instruction for the autonomous mobile device; wherein the pre-trained navigation model is a reinforcement learning model trained according to a reward function, and an output value of the reward function is obtained according to a target completion reward value, a collision reward value, a direction reward value and an obstacle depth distance reward value; and the action indicated by the action control instruction is executed, and the target point is navigated to. According to the method, indoor navigation can be accurately performed in the case that prior information about a scene map is lacking.

Description

Technical Field

[0001] This application relates to the field of navigation technology, and in particular to an indoor navigation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Navigation is a critical task for autonomous mobile devices such as robots, drones, and self-driving cars. Ensuring high-precision indoor navigation for these devices is a significant challenge. Common navigation methods include inertial navigation systems, visual navigation, lidar navigation, ultrasonic navigation, and map-based path planning. Ultrasonic navigation measures obstacle distance by emitting and detecting sound waves, but its detection range is limited and its accuracy is low, making it suitable only for short-range indoor navigation and obstacle avoidance. Map-based path planning involves using pre-built environmental maps, such as A* algorithms or Dijkstra's algorithm, to generate optimal obstacle avoidance paths. However, this method only works with environments that have pre-built maps; without prior information about the scene map, accurate indoor navigation is impossible. Summary of the Invention

[0003] Therefore, it is necessary to address the aforementioned technical problems by providing an indoor navigation method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can accurately perform indoor navigation in the absence of prior information about scene maps.

[0004] Firstly, this application provides an indoor navigation method, including:

[0005] The system acquires environmental and positioning data of an autonomous mobile device in a real-world environment. The environmental data includes depth images captured by the depth sensor of the autonomous mobile device. The positioning data includes the location data of the autonomous mobile device, the heading angle of the autonomous mobile device, and the location data of the target point.

[0006] The environmental data and the positioning data are input into a pre-trained navigation model to generate motion control commands for the autonomous mobile device; wherein, the pre-trained navigation model is a reinforcement learning model trained according to a reward function, and the output value of the reward function is obtained based on the target completion reward value, collision reward value, direction reward value, and obstacle depth and distance reward value;

[0007] Execute the action indicated by the motion control command and navigate to the target point.

[0008] In one embodiment, before inputting the environmental data and the positioning data into a pre-trained navigation model to obtain motion control commands for the autonomous mobile device, the method further includes:

[0009] Acquire sample environment data and sample positioning data of the intelligent agent in a virtual environment; the sample environment data includes sample depth images collected by the depth sensor of the intelligent agent; the sample positioning data includes the position data of the intelligent agent, the heading angle of the intelligent agent, and the position data of the sample target point; the intelligent agent is a virtual model corresponding to the autonomous mobile device in the virtual environment;

[0010] Feature extraction is performed on the sample depth image to obtain a feature vector;

[0011] The feature vector and the sample positioning data are input into the navigation model to be trained to generate sample action control commands for the intelligent agent;

[0012] When the agent executes a corresponding action based on the sample action control command, a reward function is determined according to the target completion reward value, the collision reward value, the direction reward value, and the obstacle depth and distance reward value, and the target function constructed based on the reward function is optimized to obtain the pre-trained navigation model.

[0013] In one embodiment, the step of extracting features from the sample depth image to obtain a feature vector includes:

[0014] The sample depth image is input into a residual network for feature extraction to obtain a feature vector;

[0015] The residual network consists of residual blocks containing multiple convolutional layers. Among these multiple convolutional layers, the output of any convolutional layer is allowed to bypass at least one intermediate convolutional layer directly connected to it and be directly connected to the input of the subsequent convolutional layer.

[0016] In one embodiment, prior to optimizing the objective function constructed based on the reward function, the method further includes:

[0017] The first reward value is determined by multiplying the policy ratio by the reward function; the policy ratio represents the probability ratio of the new and old policies to select the same action in a given state; the policy is the probability distribution of the agent selecting an action in a given state.

[0018] The strategy ratio is restricted to a preset range to obtain a pruned strategy ratio, and a second reward value is determined based on the product of the pruned strategy ratio and the reward function.

[0019] The objective function is determined by calculating the expected value of the minimum of the first reward value and the second reward value.

[0020] In one embodiment, before determining the reward function based on the target completion reward value, the collision reward value, the direction reward value, and the obstacle depth distance reward value, the method further includes:

[0021] Calculate the first distance and the second distance corresponding to the agent; the first distance is the distance between the previous position of the agent and the sample target point, and the second distance is the distance between the current position of the agent and the sample target point;

[0022] The directional reward value is determined based on the comparison between the first distance and the second distance and the movement distance of the agent; the movement distance is the distance between the previous position and the current position.

[0023] In one embodiment, before determining the reward function based on the target completion reward value, the collision reward value, the direction reward value, and the obstacle depth distance reward value, the method further includes:

[0024] Calculate the depth distance between the agent and obstacles in the virtual environment;

[0025] If the depth distance is greater than or equal to the first boundary value, the obstacle depth distance bonus value is determined to be the baseline bonus value;

[0026] If the depth distance is greater than or equal to the second boundary value and less than the first boundary value, the obstacle depth distance bonus value is determined as the first negative bonus value; the first negative bonus value is the negative of the target exponential function, the exponential part of the target exponential function is determined according to the distance mapping parameter, and the distance mapping parameter increases as the depth distance decreases;

[0027] If the depth distance is greater than or equal to the minimum allowable distance and less than the second boundary value, the obstacle depth distance bonus value is determined as the second negative bonus value; the second negative bonus value is a preset negative fixed value.

[0028] Secondly, this application also provides an indoor navigation device, comprising:

[0029] The acquisition module is used to acquire environmental data and positioning data of the autonomous mobile device in a real environment; the environmental data includes depth images collected by the depth sensor of the autonomous mobile device; the positioning data includes the position data of the autonomous mobile device, the heading angle of the autonomous mobile device, and the position data of the target point.

[0030] A generation module is used to input the environmental data and the positioning data into a pre-trained navigation model to generate motion control commands for the autonomous mobile device; wherein the pre-trained navigation model is a reinforcement learning model trained according to a reward function, and the output value of the reward function is obtained based on the target completion reward value, collision reward value, direction reward value, and obstacle depth and distance reward value;

[0031] A navigation module is used to navigate to the target point according to the motion control command.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0033] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0034] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0035] The aforementioned indoor navigation method, device, computer equipment, computer-readable storage medium, and computer program product acquire environmental data and positioning data of an autonomous mobile device in a real environment. The environmental data includes depth images acquired by the autonomous mobile device's depth sensor, and the positioning data includes the autonomous mobile device's position data, heading angle, and target point position data. The environmental data and positioning data are input into a pre-trained navigation model to generate motion control commands for the autonomous mobile device. The pre-trained navigation model is a reinforcement learning model trained based on a reward function, the output value of which is obtained based on target completion reward value, collision reward value, direction reward value, and obstacle depth and distance reward value. The system executes the actions indicated by the motion control commands and navigates to the target point. By acquiring depth images, obstacles and dynamic environmental changes can be accurately identified. Furthermore, by combining location data and multi-source positioning data such as heading angles, a comprehensive environmental perception system can be constructed. Meanwhile, the reinforcement learning-based navigation model can achieve adaptive decision-making through a newly designed reward function based on multi-dimensional reward values, enabling target point navigation tasks in indoor environments without prior maps. This significantly improves the environmental adaptability and robustness of indoor navigation, allowing for accurate indoor navigation even in the absence of prior information about scene maps. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an application environment diagram of an indoor navigation method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating an indoor navigation method in one embodiment;

[0039] Figure 3 This is a logic diagram of an indoor navigation method in one embodiment;

[0040] Figure 4 This is a structural diagram of a residual network in one embodiment;

[0041] Figure 5 This is a structural block diagram of an indoor navigation device in one embodiment;

[0042] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] Among related technologies, deep reinforcement learning is developing rapidly. Reinforcement learning enables agents to interact with their environment to solve various complex decision-making tasks. As one of the most important tasks in autonomous mobility, goal-point navigation requires robots to explore their surroundings with high accuracy and avoid obstacles even without a pre-existing map. Based on reinforcement learning, agents can be trained in virtual environments that simulate real-world scenarios, while striving to narrow the gap between the virtual and real environments. Once the optimal reinforcement learning model is obtained, methods such as sim2real (a transfer learning model from simulation to reality) are used to apply the trained policy to the robot in the real world.

[0045] The indoor navigation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the autonomous mobile device 102 acquires environmental and positioning data in the real environment. The environmental data includes depth images captured by the autonomous mobile device's depth sensor. The positioning data includes the autonomous mobile device's position data, heading angle, and target point position data. The autonomous mobile device 102 inputs the environmental and positioning data into a pre-trained navigation model to generate motion control commands for the autonomous mobile device. The pre-trained navigation model is a reinforcement learning model trained based on a reward function, the output of which is obtained based on target completion reward, collision reward, direction reward, and obstacle depth / distance reward. The autonomous mobile device 102 executes the actions indicated by the motion control commands and navigates to the target point.

[0046] The autonomous mobile device 102 can refer to a device with autonomous mobility capabilities, such as a robot, drone, or self-driving car, capable of perceiving, making decisions, and performing navigation tasks in the environment. Optionally, the autonomous mobile device 102 can generate sample environmental data and sample positioning data, and train the navigation model to obtain a pre-trained navigation model; or, the autonomous mobile device 102 can obtain the sample environmental data and sample positioning data required for training from the terminal 104; or, the autonomous mobile device 102 can also obtain the pre-trained navigation model from the terminal 104. The terminal 104 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.

[0047] In one exemplary embodiment, such as Figure 2 As shown, an indoor navigation method is provided, which can be applied to... Figure 1 Taking the autonomous mobile device 102 as an example, the explanation includes:

[0048] Step S202: Obtain environmental data and location data of the autonomous mobile device in the real environment.

[0049] Specifically, environmental data refers to the surrounding environment information acquired by autonomous mobile devices through sensors, used for navigation decisions. This environmental data includes depth images captured by the autonomous mobile device's depth sensor. Depth sensors can be sensors capable of measuring object distances, such as binocular cameras, structured light cameras, and Time-of-Flight (TOF) cameras. Depth sensors output depth images containing object distance information, and these images can include three-dimensional environmental data such as the geometry and spatial distribution of obstacles. High-resolution depth images allow for real-time perception of dynamic environmental changes, improving the real-time performance and adaptability of navigation.

[0050] Specifically, positioning data refers to the position and orientation information of the autonomous mobile device itself and the target point, used to determine the current location and navigation target. Positioning data includes the autonomous mobile device's position data, its heading angle, and the target point's position data. Optionally, the position data can be latitude and longitude coordinates provided by the Global Positioning System (GPS), used to determine the absolute position of the autonomous mobile device and the target point; the heading angle is the autonomous mobile device's current direction of motion (e.g., the deflection angle relative to true north), used to calculate the optimal path direction towards the target point. Optionally, the heading angle can be collected by sensors such as a compass, inertial measurement unit (IMU), or visual odometry. In practical implementation, GPS coordinates provide a global position reference, establishing the approximate orientation of the target point even in unknown environments; the heading angle is used to correct local motion direction, and combined with GPS data, it can alleviate positioning drift problems caused by weakened GPS signals in indoor scenarios (e.g., calculating short-term movement trajectories using the heading angle). Optionally, positioning data may also include position data collected by sensors such as infrared sensors and ultrasonic sensors.

[0051] In one embodiment, the autonomous mobile device can detect the signal-to-noise ratio in the environment. If the magnetic field interference intensity is determined to be higher than the intensity threshold based on the signal-to-noise ratio, the heading angle is obtained by combining the IMU and the visual odometry. If the magnetic field interference intensity is determined to be lower than the intensity threshold based on the signal-to-noise ratio, the heading angle is obtained by using a compass.

[0052] Step S204: Input environmental data and positioning data into the pre-trained navigation model to generate motion control commands for the autonomous mobile device.

[0053] Among them, the pre-trained navigation model is a machine learning model that has been trained with a large amount of simulated or real environment data and positioning data before deployment. It can be used to map the input environmental data (depth image) and positioning data (position data, heading angle, etc.) into motion control commands.

[0054] Reinforcement learning models use a reward function to determine the merit of an action performed in a given state. The learning process involves changing the policy for performing actions based on reward signals, ultimately forming a policy that maximizes the reward. It is a label-free learning process. Reinforcement learning mainly consists of an agent, an environment, a state, an action, and a reward. After the agent performs an action, the environment transitions to a new state and provides a reward signal (positive or negative) for this new state. Subsequently, based on the new state and the reward from the environment, the agent performs a new action according to a certain policy. This process represents the interaction between the agent and the environment through state, action, and reward.

[0055] The pre-trained navigation model is a reinforcement learning model trained based on a reward function. The output value of the reward function is obtained based on the target completion reward value, collision reward value, direction reward value, and obstacle depth and distance reward value.

[0056] The reward function is a mathematical function used to quantify the performance of the autonomous mobile device after each action. The output of the reward function can be determined based on the target completion reward value, collision reward value, direction reward value, and obstacle depth / distance reward value. The target completion reward value can be a positive reward for shortening the distance to the target point, a positive reward for reaching the target point, or a positive reward for the distance to the target point being less than a distance threshold. The collision reward value can be a negative penalty imposed when a collision is detected or when approaching an obstacle. The direction reward value can be a positive reward for moving in the correct direction and a negative penalty for moving in the wrong direction; the direction can be determined by pointing towards the direction closest to the target point. The obstacle depth / distance reward value refers to the reward value based on the depth distance to the obstacle; in other words, it is a dynamically adjusted reward value based on the distance to the obstacle. For example, the closer the depth distance to the obstacle, the greater the negative penalty.

[0057] The motion control command may include a motion strategy that maximizes the expected reward based on the current state, such as moving forward, turning left, turning right, and stopping. The motion control command may also be a motion parameterized command, including a range of forward distances, a range of left and right turning angles, and a specific speed.

[0058] As an example, the output value of the reward function can be equal to the sum of the target completion reward, collision reward, orientation reward, and obstacle depth and distance reward.

[0059] As another example, the output of the reward function can be equal to the weighted sum of the target completion reward, collision reward, orientation reward, and obstacle depth and distance reward.

[0060] Step S206: Execute the action indicated by the motion control command and navigate to the target point.

[0061] The motion control commands can include controlling the distance the autonomous mobile device moves forward, backward, the turning angle to the left or right, and stopping. The autonomous mobile device can convert the motion control commands output from the navigation model into low-level control signals, such as distributing the commands to its motor controller via ROS (Robot Operating System) or the CAN bus protocol. The autonomous mobile device continuously executes the motion control commands and corrects its trajectory in real time until it reaches the target point.

[0062] Therefore, reinforcement learning algorithms can be used to achieve high-precision target point navigation in indoor environments without prior maps. Target point navigation emphasizes perception and obstacle avoidance. Autonomous mobile devices can navigate from a starting point to a designated target point, perceiving the surrounding environment and autonomously planning paths to reach the final goal.

[0063] In the aforementioned indoor navigation method, environmental and positioning data of the autonomous mobile device in the real environment are acquired. The environmental data includes depth images collected by the autonomous mobile device's depth sensor, and the positioning data includes the autonomous mobile device's position data, heading angle, and target point position data. The environmental and positioning data are input into a pre-trained navigation model to generate action control commands for the autonomous mobile device. The pre-trained navigation model is a reinforcement learning model trained based on a reward function, the output of which is derived from target completion reward, collision reward, direction reward, and obstacle depth / distance reward. The action control commands are executed to navigate to the target point. By acquiring depth images, obstacles and dynamic environmental changes can be accurately identified. Furthermore, by combining position data and heading angle with multi-source positioning data, a comprehensive environmental perception system can be constructed. Simultaneously, the reinforcement learning-based navigation model can achieve adaptive decision-making through a newly designed reward function based on multi-dimensional reward values, enabling target point navigation in indoor environments without prior maps. This significantly improves the environmental adaptability and robustness of indoor navigation, allowing for accurate indoor navigation even in the absence of prior information about the scene map.

[0064] In another embodiment, before inputting environmental data and positioning data into a pre-trained navigation model to obtain action control commands for the autonomous mobile device, the method further includes: acquiring sample environmental data and sample positioning data of the agent in a virtual environment; the sample environmental data includes sample depth images collected by the agent's depth sensor; the sample positioning data includes the agent's position data, the agent's heading angle, and the position data of the sample target point; the agent is a virtual model corresponding to the autonomous mobile device in the virtual environment; extracting features from the sample depth images to obtain feature vectors; inputting the feature vectors and sample positioning data into the navigation model to be trained to generate sample action control commands for the agent; when the agent performs corresponding actions based on the sample action control commands, determining a reward function based on the target completion reward value, collision reward value, direction reward value, and obstacle depth and distance reward value, and optimizing the target function constructed based on the reward function to obtain a pre-trained navigation model.

[0065] For the convenience of those skilled in the art, Figure 3 An example logic diagram of an indoor navigation method is provided.

[0066] During training, the agent can be a mobile robot model equipped with a differential drive chassis and virtual sensors, such as the Fetch robot model. The agent is used for training in a virtual environment to help the autonomous mobile device achieve autonomous obstacle avoidance and target point navigation. It should be noted that the agent is a virtual model corresponding to the autonomous mobile device in the virtual environment. This means that the virtual model performs the same navigation tasks (obstacle avoidance, path planning) as the real autonomous mobile device. However, the virtual model does not need to replicate the type (robot, drone, etc.), mechanical structure, drive type, etc. of the autonomous mobile device. The virtual model is only used to carry the navigation strategy, so it can be adapted to multiple types of devices.

[0067] like Figure 3 As shown, the training framework includes a virtual engine (Unreal Engine, UE), a SpearSim simulator, a Python client, sensors, deep reinforcement learning components, and a reinforcement learning model.

[0068] The virtual engine is used to construct virtual environments, which can be 3D scenes of static obstacles (such as walls and furniture) and dynamic interactive objects (such as moving obstacles). Intelligent agents can be trained within this virtual environment.

[0069] The SpearsSim platform is used to enable bidirectional communication between Unreal Engine 5 (UE5) and the Python API. The SpearsSim platform can transmit GPS data, compass data, and depth images to the Python side and receive action commands from the Python side to drive the movement of the intelligent agent.

[0070] The Python side is used to handle deep reinforcement learning algorithms and sensor inputs. Sensor inputs can include sample environment data and sample localization data. Sample environment data can include sample depth images acquired by the agent's depth sensor. Sample localization data can include the agent's position data in the virtual environment (e.g., GPS coordinates), the agent's heading angle in the virtual environment (e.g., compass data), and the position data of sample target points. Sample target points can refer to the global 3D coordinates in the virtual environment during training. Optionally, sample target points can be randomly generated and reset in each training cycle, or they can be fixed points set in the virtual environment, such as a room corner or the origin of the environment.

[0071] The deep reinforcement learning component can extract features from depth images using deep learning networks, such as residual networks, to obtain feature vectors. Simultaneously, a reinforcement learning model can be built using the Stable Baselines3 framework and trained using the Policy Proximity Optimization (PPO) algorithm. The SpearSim simulator allows real-time access to sensor inputs and transmits motion control commands to the agent in real time, enabling target point navigation based on deep reinforcement learning.

[0072] Specifically, reinforcement learning maximizes reward signals through interaction with the environment. In reinforcement learning, an agent performs a series of actions in the environment to receive feedback (reward or punishment), adjusting its policy accordingly to obtain as much long-term cumulative reward as possible. Reinforcement learning consists of five key components: agent, environment, state, reward, and action.

[0073] The environment takes the agent's current state and actions as input and outputs the reward received by the agent and the state after the action is performed. The training task aims to train the agent to avoid obstacles and reach a target point in an indoor environment with many obstacles. A virtual environment is constructed using sim2real, and it is required that the virtual environment be as similar to the real environment as possible. Optionally, the environment can be rendered using Unreal Engine 5, and the scene can be a virtual indoor room. To enhance the realism of the scene, objects such as dining tables and chairs are placed inside, and the agent needs to navigate the scene while avoiding static obstacles such as coffee tables and sofas.

[0074] Here, state refers to the state of the environment at a specific moment. The agent perceives the current state to make the next decision and perform an action, and the state describes all relevant information in the environment at that time. Optionally, the state can be determined by a depth image captured by the agent's front-facing camera. The depth image can be the result of rendering in Unreal Engine 5, and sample depth images at angles unaffected by factors such as lighting can be used as state input.

[0075] In this context, an action is an operation performed by the agent. The agent selects an action based on the current state, aiming to maximize future cumulative rewards. Each environment has a corresponding action space. The action space can consist of four actions: forward, left turn, right turn, and stop. Since these actions are controlled by differential actuation of the agent's wheels, a continuous action space can be used. Considering the physical properties of the virtual environment, such as friction, each action is normalized. Furthermore, to prevent the agent from moving too fast, specific ranges are set for actions: forward range is 0 to 0.6, left and right turn range is 0 to 0.3, and stop range is 0.

[0076] In this context, the agent is an entity that performs actions in the environment, receives feedback, and adjusts its strategy based on experience; it can also be called an intelligent agent. The intelligent agent learns to take appropriate actions in different states to maximize long-term cumulative rewards. The intelligent agent follows a cycle of perceiving the environment, making decisions, performing actions, receiving feedback, obtaining rewards, and updating its strategy, continuously learning and optimizing. Since the training task is to achieve target point navigation, to make the training results more intuitive, the robotic arm of the Fetch robot model was removed, leaving only the mobile base as the agent.

[0077] In reinforcement learning training, rewards play a crucial role, providing the agent with learning direction and decision-making basis. Rewards are typically feedback signals provided by the environment after the agent takes an action. A well-designed reward function can effectively guide the agent to learn correct behaviors, while an unreasonable reward function may lead to undesirable behaviors or poor learning outcomes. When the agent completes a task, the environment provides positive rewards to encourage it to repeat these actions. When the agent takes inappropriate or incorrect behaviors, the environment provides negative rewards to prevent it from repeating these behaviors. Rewards guide the agent to choose behaviors that maximize long-term cumulative rewards.

[0078] visible, Figure 3 A novel point navigation task system was constructed based on the SpearSim platform, integrating scene rendering and reinforcement learning training. By designing the action space, input image information, and reward structure, embedded intelligent point navigation tasks can be developed, achieving more efficient navigation.

[0079] In another embodiment, before optimizing the objective function constructed based on the reward function, the method further includes: determining a first reward value based on the product of the policy ratio and the reward function; the policy ratio represents the probability ratio of the new and old policies to select the same action in a given state; the policy is the probability distribution of the agent selecting an action in a given state; limiting the policy ratio to a preset range to obtain a pruned policy ratio, and determining a second reward value based on the product of the pruned policy ratio and the reward function; and calculating the expected value of the minimum of the first reward value and the second reward value to determine the objective function.

[0080] Specifically, the core of reinforcement learning algorithms is to continuously optimize the agent's decision-making strategy through interaction with the environment, enabling it to maximize long-term cumulative rewards in a given environment. Reinforcement learning algorithms can be divided into value-based algorithms, policy-based proximity policy optimization (PRO) algorithms, and hybrid methods (Actor-Critic).

[0081] The PPO algorithm is an improved version of the TRPO algorithm, offering a simpler method and higher learning efficiency. TRPO addresses instability by controlling the magnitude of policy updates, but its implementation is complex, requiring the solution of a quadratic programming problem and making it difficult to scale to large problems. PPO, on the other hand, limits the distance between the new and old policies during the update process to avoid unstable behavior caused by excessive policy changes. PPO provides two implementations: Clipped PPO and Adaptive KL Penalized PPO. Optionally, the Clipped PPO algorithm can be used to construct the objective function of the reinforcement learning model, improving training stability through policy ratio clipping.

[0082] Wherein, the strategy is a specific state s t Choose action a t The probability distribution of can be denoted as . old strategy Given the policy parameters from the previous iteration, the new policy... These are the current strategy parameters to be optimized.

[0083] The strategy ratio is The policy ratio represents the probability ratio of choosing the same action under a given state between the new and old policies.

[0084] .

[0085] The strategy ratio can be used to measure the difference between the old and new strategies and to apply constraints to strategy updates during the optimization process.

[0086] Limiting the strategy ratio to a preset range, i.e., performing strategy ratio clipping, can restrict the strategy ratio to a specific range. Inside, optional. It can be equal to 0.2. By introducing a limit, excessive policy updates can be prevented, keeping the policy ratio within a reasonable range.

[0087] For example, the objective function can be expressed as:

[0088] ;

[0089] in, To obtain the expected value, This is a shearing function used to adjust the policy ratio. Limited to the range Within, the cropping strategy ratio is obtained. Here is the reward function, used to represent a given state s. t Next action a t The quality of the policy. The first reward value is equal to the product of the policy ratio and the reward function, expressed as: The second reward value is equal to the product of the pruning policy ratio and the reward function, expressed as: Calculate the expected value of the minimum between the first and second reward values ​​to determine the objective function. .

[0090] The technical solution of the above embodiments utilizes the PPO algorithm in reinforcement learning to assist the robot in avoiding obstacles and reaching a designated point indoors. When the policy change is small, the policy can be freely optimized. However, when the policy change is large, the pruning function limits the increase or decrease of the ratio to prevent excessive policy updates that may cause a sudden drop in performance. The pruning mechanism forces the policy update magnitude to be within a controllable range, avoiding performance fluctuations or crashes caused by excessively large single updates.

[0091] In other embodiments, the policy ratio is limited to a preset ratio range, which can be switched according to different types of actions (such as movement and turning). For example, the preset ratio range may include a first preset ratio range corresponding to the agent performing a movement action and a second preset ratio range corresponding to the agent performing a turning action. The first preset ratio range is greater than the second preset ratio range, thereby allowing a larger update when the agent performs a movement action and strictly limiting the update range when the agent performs a turning action, thus finely controlling the update risk of different actions, which is suitable for complex navigation tasks.

[0092] In another embodiment, before determining the reward function based on the target completion reward value, collision reward value, directional reward value, and obstacle depth distance reward value, the method further includes: calculating a first distance and a second distance corresponding to the agent; the first distance is the distance between the agent's previous position and the sample target point, and the second distance is the distance between the agent's current position and the sample target point; determining the directional reward value based on the comparison result between the first distance and the second distance and the agent's movement distance; the movement distance is the distance between the previous position and the current position.

[0093] Assume the coordinates of the target point of the sample are The coordinates of the previous position are The coordinates of the current position are .

[0094] The first distance can be expressed as:

[0095] .

[0096] The second distance can be expressed as:

[0097] .

[0098] The distance traveled can refer to the Euclidean distance from the previous position to the current position, which can be expressed as:

[0099] .

[0100] As an example, an autonomous mobile device can determine a positive reward based on a comparison between a first distance and a second distance. If the first distance is greater than the second distance, the agent is approaching the target point and receives a positive reward. The directional reward value can be positive, and its specific value can be determined by the product of the movement distance and the positive reward coefficient. Conversely, if the first distance is less than the second distance, the agent is moving away from the target point and receives a negative reward. The directional reward value can be negative, and its specific value can be determined by the product of the movement distance and the negative reward coefficient. Since the agent may need to move backward due to obstacles during navigation, the coefficients for positive and negative rewards are different. Optionally, the positive reward coefficient can be greater than the negative reward coefficient; for example, the positive reward coefficient could be 0.07, and the negative reward coefficient 0.05. A target function constructed based on the directional reward value can be used to ensure the agent moves towards the target point, i.e., moves in the correct direction.

[0101] As an example, the positive and negative reward coefficients can be dynamically adjusted based on environmental complexity. The positive reward coefficient can be determined by multiplying the base positive reward coefficient by a first adjustment coefficient. The first adjustment coefficient can be equal to the ratio of obstacle density to maximum density plus 1. Obstacle density is calculated using a depth image; in scenarios with high obstacle density, the reward is greater if the player moves in the correct direction. The negative reward coefficient can be determined by multiplying the base negative reward coefficient by a second adjustment coefficient. The second adjustment coefficient can be equal to the ratio of second distance to first distance plus 1. The ratio of second distance to first distance characterizes the difference in distance from the target point between previous and subsequent positions; the greater the difference, the greater the penalty.

[0102] The technical solution of the above embodiments can improve the accuracy of indoor navigation by finely evaluating the effectiveness of the navigation direction through the distance change during the movement of the intelligent agent.

[0103] In another embodiment, before determining the reward function based on the target completion reward value, collision reward value, direction reward value, and obstacle depth distance reward value, the method further includes: calculating the depth distance between the agent and obstacles in the virtual environment; determining the obstacle depth distance reward value as a baseline reward value if the depth distance is greater than or equal to a first boundary value; determining the obstacle depth distance reward value as a first negative reward value if the depth distance is greater than or equal to a second boundary value and less than the first boundary value; the first negative reward value is a negative of the target exponential function, the exponential part of which is determined based on a distance mapping parameter, which increases as the depth distance decreases; and determining the obstacle depth distance reward value as a second negative reward value if the depth distance is greater than or equal to the minimum allowable distance and less than the second boundary value; the second negative reward value is a preset negative value.

[0104] In practice, since indoor environments generally have many obstacles, rewards related to depth distance can be designed to achieve obstacle avoidance. First, depth estimation is performed using depth images to determine the current distance between the agent and the obstacle in front, thus obtaining the depth distance.

[0105] The first boundary value d1 can be a safety distance threshold, such as 1 meter. Exceeding this distance can be considered as an obstacle-free risk. Optionally, if the depth distance is greater than or equal to the first boundary value, the baseline reward value can be 0 or a value close to 0, that is, no penalty is imposed.

[0106] Here, the second boundary value d2 can be a warning distance threshold, such as 10 cm to 50 cm, triggering a moderate penalty. When the depth distance is greater than or equal to the second boundary value but less than the first boundary value, the obstacle depth distance reward value is determined as the first negative reward value. The first negative reward value is the negative of the target exponential function, where the exponent is determined based on the distance mapping parameters, which increase as the depth distance decreases. For example, the first negative reward value can be expressed as... , is the objective exponential function The negative numbers, where, Custom parameters can be defined, ranging from 2.5 to 2.91.

[0107] Optional, distance mapping parameters It can be represented as:

[0108] ;

[0109] It can be seen that when the depth distance When the value decreases, exponential growth accelerates.

[0110] Among them, the minimum allowable distance is the danger distance threshold. For example, a value of 0 or close to 0 can trigger an emergency penalty. The obstacle depth distance reward value is determined as the second negative reward value, and the second negative reward value is a preset negative constant value. Moreover, the absolute value of the second negative reward value can be greater than the absolute value of the first negative reward value. For example, the second negative reward value can be equal to -C, where C can be equal to 50 - 80, and a high penalty is directly imposed.

[0111] Through the technical solution of the above embodiment, a new reward mechanism is designed based on the change in the depth distance between the agent and the obstacle during movement, which can improve the training accuracy of the reinforcement learning model, thereby improving the accuracy of indoor navigation.

[0112] In some other embodiments, the target completion reward value can be directly given as a preset positive constant value, such as 500 - 1000, the highest reward value; the collision reward can be given as a preset negative constant value, such as -100 - -200, to punish the agent for colliding with obstacles in the environment.

[0113] Through the technical solution of the above embodiment, through exponential non - linear penalty and hierarchical safety boundaries, a refined response to the distance of obstacles is achieved, which can improve the training accuracy of the reinforcement learning model, thereby improving the accuracy of indoor navigation.

[0114] In one of the embodiments, the reward function can be expressed as shown in Table 1:

[0115] Table 1

[0116]

[0117] It can be seen that this reward calculation framework consists of four parts: reaching the target, collision, direction, and depth distance from the obstacle. When the agent reaches the target point, it will obtain the highest reward of 1000. If the agent collides with an object in the environment, it will obtain a reward of -100.

[0118] The direction reward value can be used to ensure that the agent moves towards the target point, that is, moves in the correct direction. When old_distance > new_distance, a positive reward of 5 + distance_travelled * 0.07 can be obtained, and the positive reward coefficient is 0.07. When old_distance < new_distance, a negative reward of -(5 + distance_travelled * 0.05) can be obtained, and the negative reward coefficient is 0.05.

[0119] The obstacle depth distance reward is used to ensure the agent can avoid obstacles. If the depth distance to the obstacle is greater than 1 meter, the agent will not receive any reward. If the distance is between 0 and 10 centimeters, the agent will receive a reward of -80. To ensure that the reward increases non-linearly as the agent approaches the obstacle, a formula can be used... The reward value is calculated such that as the input value increases linearly, the reward increases exponentially. When the depth distance is between 10 cm and 1 m, it is mapped to the range δ∈[1,5]. As the depth distance decreases, The value gradually increases. (Coefficient) It is a custom parameter, for example, 2.91.

[0120] Optional, It can be represented as:

[0121] ;

[0122] This formula can be used to express how the depth distance decreases linearly from 1 meter to 10 centimeters. It can be increased from 1 to 5.

[0123] As another example, the output of the reward function can be equal to the weighted sum of the target completion reward, collision reward, orientation reward, and obstacle depth / distance reward. Furthermore, the weights of each reward value can be adaptively adjusted according to the scene; for example, areas with dense dynamic obstacles require higher safety, while open areas prioritize efficiency. Specifically, the reward function can include a first reward function and a second reward function. The first reward function performs scene semantic analysis on the sample depth image to obtain semantic segmentation results. If the semantic segmentation results indicate that the current scene is an obstacle-dense area, the weight of the collision reward is adjusted to be higher than the target completion reward, and the weighted sum of the target completion reward, collision reward, orientation reward, and obstacle depth / distance reward is calculated based on the adjusted weights. If the semantic segmentation results indicate that the current scene is an open area, the weight of the target completion reward is adjusted to be higher than the weight of the collision reward, and the weighted sum of the target completion reward, collision reward, orientation reward, and obstacle depth / distance reward is calculated based on the adjusted weights. When the autonomous mobile device is actually running, it can also perform scene semantic analysis on real-time depth images to obtain semantic segmentation results. If the semantic segmentation results indicate that the current scene is a densely obstacle-filled area, a reinforcement learning model trained based on the first reward function is used; if the semantic segmentation results indicate that the current scene is an open area, a reinforcement learning model trained based on the second reward function is used.

[0124] In other embodiments, feature extraction is performed on the sample depth image to obtain a feature vector, including: inputting the sample depth image into a residual network for feature extraction to obtain a feature vector; wherein the residual network is composed of residual blocks containing multiple convolutional layers, and the output of any convolutional layer is allowed to bypass at least one intermediate convolutional layer directly connected to it and be directly connected to the input of the subsequent convolutional layer.

[0125] During navigation, depth images are used as input information. However, directly inputting images into the model not only prevents the model from extracting key information from the images but also leads to insufficient memory. Therefore, residual networks (such as ResNet18) can be used as deep learning networks for feature extraction.

[0126] In traditional deep learning networks, the vanishing and exploding gradient problems become more severe as the number of layers increases, making training more difficult. This residual network addresses these degradation issues by introducing skip connections. The residual network allows the output of one layer to bypass one or more layers and directly connect to the input of subsequent layers. This ensures that even if some layers do not perform any transformations, information from previous layers can still be passed on, thus preventing excessive gradient loss.

[0127] Specifically, a residual block contains multiple convolutional layers (e.g., 3 layers: Conv1, Conv2, Conv3), but allows the output of any convolutional layer to bypass subsequent layers and directly propagate to more distant layers. For example, Conv1 -> (Conv2 can be skipped) -> Conv3. In the implementation, during training, a learnable gating mechanism automatically determines whether to skip intermediate layers. Each convolutional layer can be assigned a corresponding gating coefficient; a gating coefficient of 1 retains the layer, while a gating coefficient of 0 skips the layer.

[0128] For the convenience of those skilled in the art, Figure 4 An exemplary structural diagram of a residual network is provided, illustrating the core design of a residual block. The residual network can learn the residual mapping F(x). By introducing an identity mapping path, the original input x can bypass the intermediate processing layer and directly participate in feature fusion.

[0129] In practical applications, assuming the input depth image is set to a size of 128x128x1, after passing through the residual network, an output vector of size 1n can be generated. After a series of controlled experiments, it can be determined that the dimension of the output vector is 1x4.

[0130] The technical solution of the above embodiments extracts scene information through a deep learning network and introduces an improved residual network, which can optimize the feature extraction process of depth images through dynamic skip connections, solve the degradation problem in deep learning network training, and improve the training accuracy of the navigation model.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0132] Based on the same inventive concept, this application also provides an indoor navigation device for implementing the indoor navigation method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more indoor navigation device embodiments provided below can be found in the limitations of the indoor navigation method described above, and will not be repeated here.

[0133] In one exemplary embodiment, such as Figure 5 As shown, an indoor navigation device is provided, comprising:

[0134] The acquisition module 510 is used to acquire environmental data and positioning data of the autonomous mobile device in a real environment; the environmental data includes depth images collected by the depth sensor of the autonomous mobile device; the positioning data includes the position data of the autonomous mobile device, the heading angle of the autonomous mobile device, and the position data of the target point.

[0135] The generation module 520 is used to input the environmental data and the positioning data into a pre-trained navigation model to generate motion control commands for the autonomous mobile device; wherein the pre-trained navigation model is a reinforcement learning model trained according to a reward function, and the output value of the reward function is obtained based on the target completion reward value, collision reward value, direction reward value and obstacle depth and distance reward value.

[0136] The navigation module 530 is used to execute the actions indicated by the action control command and navigate to the target point.

[0137] In one embodiment, the indoor navigation device further includes a training module, specifically used to acquire sample environment data and sample positioning data of the agent in a virtual environment; the sample environment data includes sample depth images collected by the depth sensor of the agent; the sample positioning data includes the position data of the agent, the heading angle of the agent, and the position data of the sample target point; the agent is a virtual model corresponding to the autonomous mobile device in the virtual environment; features are extracted from the sample depth images to obtain feature vectors; the feature vectors and the sample positioning data are input into the navigation model to be trained to generate sample action control commands for the agent; when the agent performs corresponding actions based on the sample action control commands, a reward function is determined according to the target completion reward value, the collision reward value, the direction reward value, and the obstacle depth distance reward value, and the target function constructed according to the reward function is optimized to obtain the pre-trained navigation model.

[0138] In one embodiment, the training module is specifically used to input the sample depth image into a residual network for feature extraction to obtain a feature vector; wherein the residual network is composed of residual blocks containing multiple convolutional layers, and the output of any convolutional layer is allowed to bypass at least one intermediate convolutional layer directly connected to it and be directly connected to the input of the subsequent convolutional layer.

[0139] In one embodiment, the training module is specifically configured to determine a first reward value based on the product of the policy ratio and the reward function; the policy ratio represents the probability ratio of the new and old policies selecting the same action in a given state; the policy is the probability distribution of the agent selecting an action in a given state; the policy ratio is restricted to a preset range to obtain a pruned policy ratio, and a second reward value is determined based on the product of the pruned policy ratio and the reward function; the expected value of the minimum of the first reward value and the second reward value is calculated to determine the objective function.

[0140] In one embodiment, the training module is specifically used to calculate a first distance and a second distance corresponding to the agent; the first distance is the distance between the previous position of the agent and the sample target point, and the second distance is the distance between the current position of the agent and the sample target point;

[0141] The directional reward value is determined based on the comparison between the first distance and the second distance and the movement distance of the agent; the movement distance is the distance between the previous position and the current position.

[0142] In one embodiment, the training module is specifically configured to calculate the depth distance between the agent and obstacles in the virtual environment; if the depth distance is greater than or equal to a first boundary value, determine the obstacle depth distance reward value as a baseline reward value; if the depth distance is greater than or equal to a second boundary value and less than the first boundary value, determine the obstacle depth distance reward value as a first negative reward value; the first negative reward value is a negative number of a target exponential function, the exponential part of which is determined based on a distance mapping parameter, which increases as the depth distance decreases; if the depth distance is greater than or equal to a minimum allowable distance and less than the second boundary value, determine the obstacle depth distance reward value as a second negative reward value; the second negative reward value is a preset negative fixed value.

[0143] The modules in the aforementioned indoor navigation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0144] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an indoor navigation method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0145] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An indoor navigation method, characterized in that, The method includes: The system acquires environmental and positioning data of an autonomous mobile device in a real-world environment. The environmental data includes depth images captured by the depth sensor of the autonomous mobile device. The positioning data includes the location data of the autonomous mobile device, the heading angle of the autonomous mobile device, and the location data of the target point. Scene semantic analysis is performed on the depth image to obtain semantic segmentation results; When the semantic segmentation result indicates that the current scene is an area with dense obstacles, the environmental data and the positioning data are input into a pre-trained navigation model to generate action control commands for the autonomous mobile device; wherein, the pre-trained navigation model is a reinforcement learning model trained according to a first reward function, the output value of the first reward function is obtained by weighted summation of target completion reward value, collision reward value, direction reward value and obstacle depth and distance reward value, and the weight of the collision reward value is higher than that of the target completion reward value; When the semantic segmentation result indicates that the current scene is an open area, the environmental data and the positioning data are input into a pre-trained navigation model to generate action control commands for the autonomous mobile device; wherein, the pre-trained navigation model is a reinforcement learning model trained according to a second reward function, the output value of the second reward function is obtained by weighted summation of target completion reward value, collision reward value, direction reward value and obstacle depth and distance reward value, and the weight of the target completion reward value is higher than the weight of the collision reward value; The directional reward value is determined based on the comparison between the first distance and the second distance corresponding to the agent and the moving distance of the agent; the first distance is the distance between the previous position of the agent and the sample target point, the second distance is the distance between the current position of the agent and the sample target point, and the moving distance is the distance between the previous position and the current position; Wherein, the obstacle depth distance reward value is equal to the baseline reward value when the depth distance between the agent and the obstacle in the virtual environment is greater than or equal to a first boundary value; the obstacle depth distance reward value is equal to a first negative reward value when the depth distance is greater than or equal to a second boundary value and less than the first boundary value, and the first negative reward value is expressed as follows: , For custom parameters, The obstacle depth distance bonus increases as the depth distance decreases; when the depth distance is greater than or equal to the minimum allowable distance and less than the second boundary value, the obstacle depth distance bonus is equal to the second negative bonus value, which is a preset negative fixed value. Execute the action indicated by the motion control command and navigate to the target point.

2. The method according to claim 1, characterized in that, Before inputting the environmental data and the positioning data into the pre-trained navigation model to obtain motion control commands for the autonomous mobile device, the method further includes: Acquire sample environment data and sample positioning data of the intelligent agent in a virtual environment; the sample environment data includes sample depth images collected by the depth sensor of the intelligent agent; the sample positioning data includes the position data of the intelligent agent, the heading angle of the intelligent agent, and the position data of the sample target point; the intelligent agent is a virtual model corresponding to the autonomous mobile device in the virtual environment; Feature extraction is performed on the sample depth image to obtain a feature vector; The feature vector and the sample positioning data are input into the navigation model to be trained to generate sample action control commands for the intelligent agent; When the agent executes a corresponding action based on the sample action control command, a reward function is determined according to the target completion reward value, the collision reward value, the direction reward value, and the obstacle depth and distance reward value, and the target function constructed based on the reward function is optimized to obtain the pre-trained navigation model.

3. The method according to claim 2, characterized in that, The step of extracting features from the sample depth image to obtain a feature vector includes: The sample depth image is input into a residual network for feature extraction to obtain a feature vector; The residual network consists of residual blocks containing multiple convolutional layers. Among these multiple convolutional layers, the output of any convolutional layer is allowed to bypass at least one intermediate convolutional layer directly connected to it and be directly connected to the input of the subsequent convolutional layer.

4. The method according to claim 2, characterized in that, Prior to optimizing the objective function constructed based on the reward function, the method further includes: The first reward value is determined by multiplying the policy ratio by the reward function; the policy ratio represents the probability ratio of the new and old policies to select the same action in a given state; the policy is the probability distribution of the agent selecting an action in a given state. The strategy ratio is restricted to a preset range to obtain a pruned strategy ratio, and a second reward value is determined based on the product of the pruned strategy ratio and the reward function. The objective function is determined by calculating the expected value of the minimum of the first reward value and the second reward value.

5. An indoor navigation device, characterized in that, The device includes: The acquisition module is used to acquire environmental data and positioning data of the autonomous mobile device in a real environment; the environmental data includes depth images collected by the depth sensor of the autonomous mobile device; the positioning data includes the position data of the autonomous mobile device, the heading angle of the autonomous mobile device, and the position data of the target point. A generation module is used to perform scene semantic analysis on the depth image to obtain semantic segmentation results. When the semantic segmentation results indicate that the current scene is an area with dense obstacles, the environmental data and the positioning data are input into a pre-trained navigation model to generate action control commands for the autonomous mobile device. The pre-trained navigation model is a reinforcement learning model trained according to a first reward function, the output of which is a weighted sum of target completion reward, collision reward, direction reward, and obstacle depth / distance reward, with the collision reward having a higher weight than the target completion reward. When the semantic segmentation results indicate that the current scene is an open area, the environmental data and the positioning data are input into the pre-trained navigation model to generate action control commands for the autonomous mobile device. The pre-trained navigation model is a reinforcement learning model trained according to a second reward function, the output of which... The value is obtained by weighted summation of target completion reward, collision reward, directional reward, and obstacle depth distance reward, with the target completion reward having a higher weight than the collision reward. The directional reward is determined by comparing a first distance and a second distance between the agent and the agent's movement distance. The first distance is the distance between the agent's previous position and the sample target point, the second distance is the distance between the agent's current position and the sample target point, and the movement distance is the distance between the previous position and the current position. The obstacle depth distance reward is equal to the baseline reward when the depth distance between the agent and obstacles in the virtual environment is greater than or equal to a first boundary value; and equal to a first negative reward when the depth distance is greater than or equal to the second boundary value and less than the first boundary value. The first negative reward is expressed as... , For custom parameters, The obstacle depth distance bonus increases as the depth distance decreases; when the depth distance is greater than or equal to the minimum allowable distance and less than the second boundary value, the obstacle depth distance bonus is equal to the second negative bonus value, which is a preset negative fixed value. The navigation module is used to execute the actions indicated by the motion control command and navigate to the target point.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images