Methods, apparatus, media, and equipment for map-free navigation based on open 3D worlds.

Abstract

This application relates to the field of game navigation technology, specifically providing a method, apparatus, medium, and device for mapless navigation in an open 3D world. The method includes: inputting image features of the currently displayed game interface and pose vector features of an agent into a fusion network to predict the agent's next action; determining a reward value by analyzing the agent's actual position after performing the next action; the reward value includes positive and negative reward values; a positive reward value indicates that the agent is gradually approaching the task target point, while a negative reward value indicates that the agent is gradually moving away from the task target point or approaching a target obstacle; adjusting the network parameters of the fusion network based on the reward value to obtain a target fusion model; the target fusion model is used to provide action and direction navigation for the agent when it performs a game task. Embodiments of this application can achieve accurate navigation of the agent in mapless game scenarios, improving the user experience.

Description

Technical Field

[0001] This application relates to the field of game navigation technology, and more specifically, to a method, apparatus, medium, and device for mapless navigation based on an open 3D world. Background Technology

[0002] Open-world 3D games refer to video games that offer a vast game world where players can freely explore, interact, and complete tasks. The core characteristic of these games is providing players with a broad, freely explorable environment where they can freely choose their goals and paths, interact with other game characters, and engage in various activities. Currently, navigation in open-world 3D games primarily employs two methods: one is static map navigation based on relevant pathfinding algorithms, which requires preloading a pre-built complete scene map; the other is basic Reinforcement Learning (RL) algorithms, trained using DQN (Deep Q-Network) or the original PPO (Proximal Policy Optimization) algorithm, with RGB images as input data, relying on recognizing pathfinding cues in the game (such as minimap target points, suggested routes, etc.). However, both of these methods are highly dependent on the static environment, cannot adapt to dynamic obstacles in real time, have poor navigation and obstacle avoidance capabilities, cannot accurately navigate agents within tasks, and have low task completion efficiency.

[0003] Therefore, how to provide a technical solution for a highly accurate mapless navigation method based on an open 3D world has become an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of some embodiments of this application is to provide a method, apparatus, medium and device for mapless navigation in open 3D worlds. The technical solutions of the embodiments of this application can improve the accuracy of mapless navigation in open 3D world games, enabling intelligent agents to find the correct path, improving task completion efficiency and user gaming experience.

[0005] In a first aspect, some embodiments of this application provide a method for mapless navigation based on an open 3D open world, comprising: inputting image features of the game interface currently displayed on the screen and pose vector features of an agent into a fusion network to predict the next action of the agent; wherein, the image features include: object information and relative distances between objects; the next action includes: the action type and movement direction of the agent; determining a reward value by analyzing the actual position of the agent after performing the next action; wherein, the reward value includes positive reward values ​​and negative reward values; the positive reward value indicates that the agent is gradually approaching the task target point, and the negative reward value indicates that the agent is gradually moving away from the task target point or gradually approaching a target obstacle; adjusting the network parameters of the fusion network based on the reward value to obtain a target fusion model; wherein, the target fusion model is used to provide action and direction navigation for the agent when the agent performs a game task.

[0006] Some embodiments of this application use the image features of the current screen and the pose feature vector of the agent to predict the next action of the agent using a fusion model; by analyzing the actual position of the agent after performing the next action, the reward value for the next action is determined, and then the network parameters of the fusion network are adjusted by the reward value to obtain a target fusion model. This results in a target fusion model for accurate navigation of the agent, and improves the accuracy of mapless navigation in open 3D open-world games, enabling the agent to find the correct path, improving task completion efficiency and user gaming experience.

[0007] In some embodiments, before inputting the image features of the game interface currently displayed on the screen and the pose vector features of the agent into the fusion network, the method further includes: acquiring image information of the game interface currently displayed on the screen and acquiring pose information of the agent in the current screen; analyzing the image information to determine the image features in the image information; and generating the pose vector features corresponding to the pose information.

[0008] Some embodiments of this application generate image features by analyzing the image information currently displayed on the screen, and convert the pose information of the intelligent agent into pose vector features, thereby providing data support for predicting the intelligent agent's actions.

[0009] In some embodiments, the image features include: an instance segmentation map and a relative depth map; the step of analyzing the image information to determine the image features in the image information includes: inputting the image information into a pre-trained instance segmentation network to obtain the instance segmentation map; wherein the instance segmentation map represents the object information of various objects contained in the image information; inputting the image information into a pre-trained monocular depth network to obtain the relative depth map; wherein the relative depth map represents the relative distance between various objects.

[0010] Some embodiments of this application obtain instance segmentation maps and relative depth maps by inputting image information into different pre-trained instance segmentation networks and monocular depth networks, thereby achieving effective segmentation and extraction of image information.

[0011] In some embodiments, determining the reward value by analyzing the actual position of the agent after it performs the next action includes: if the relative distance between the actual position and the task target point is less than a distance threshold, then the positive reward value is generated.

[0012] Some embodiments of this application generate reward values ​​by determining the distance relationship between the actual position of the agent and the task target point, which can ensure the accuracy of the agent's movement direction.

[0013] In some embodiments, when there is a height difference between the task target point and the agent's current position, the actual position includes a horizontal position and a vertical height from the ground. The step of analyzing the actual position of the agent after performing the next action to determine the reward value includes: if the horizontal distance between the agent's horizontal position and the task target point is not less than a distance threshold, then the reward value is calculated according to a preset rule; if the horizontal distance between the agent's horizontal position and the task target point is less than a distance threshold and the vertical distance between the agent and the task target point is greater than a height threshold, then the reward value is generated in the vertical direction; if the horizontal distance between the agent's horizontal position and the task target point is less than a distance threshold and the vertical distance between the agent and the task target point is not greater than a height threshold, then the reward value is generated in the horizontal direction.

[0014] Some embodiments of this application generate different reward values ​​by analyzing the distance relationship between the agent and the task target point in the horizontal and vertical directions, which can ensure that the agent can move accurately in scenes with height differences.

[0015] In some embodiments, the reward value generated in the vertical direction is within a set reward range.

[0016] Some embodiments of this application avoid reward fluctuations by setting a reward range.

[0017] In some embodiments, the step of analyzing the actual position of the agent after performing the next action to determine the reward value includes: confirming that the distance between the actual position and the target obstacle is less than the distance between the agent's current position and the target obstacle, then generating the negative reward value; the negative reward value indicates that the agent is gradually approaching the target obstacle.

[0018] Some embodiments of this application can improve the obstacle avoidance capability of the intelligent agent by analyzing the actual position of the intelligent agent and the distance between obstacles.

[0019] Secondly, some embodiments of this application provide a device for map-free navigation based on an open 3D world, comprising: a fusion module, used to input image features of the game interface currently displayed on the screen and pose vector features of an agent into a fusion network to predict the next action of the agent; wherein, the image features include: object information and relative distances between objects; the next action includes: the action type and movement direction of the agent; a reward module, used to determine a reward value by analyzing the actual position of the agent after performing the next action; wherein, the reward value includes positive reward values ​​and negative reward values; the positive reward value indicates that the agent is gradually approaching the task target point, and the negative reward value indicates that the agent is gradually moving away from the task target point or gradually approaching a target obstacle; an optimization module, used to adjust the network parameters of the fusion network based on the reward value to obtain a target fusion model; wherein, the target fusion model is used to provide action and direction navigation for the agent when the agent performs a game task.

[0020] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0021] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0022] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 System diagrams for mapless navigation based on open 3D open world provided for some embodiments of this application;

[0025] Figure 2 One of the flowcharts for a method of mapless navigation based on an open 3D open world provided for some embodiments of this application;

[0026] Figure 3 The second flowchart of a method for mapless navigation based on an open 3D open world, provided for some embodiments of this application;

[0027] Figure 4 Block diagrams of an apparatus for mapless navigation based on an open 3D world, provided for some embodiments of this application;

[0028] Figure 5 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0029] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0030] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] In related technologies, navigation in open 3D open-world games often relies on pre-generated NavMesh or traditional reinforcement learning algorithms (such as DQN). However, existing technologies suffer from static environment dependence; that is, NavMesh needs to be pre-built and cannot adapt to dynamic obstacles in real time, and static map navigation requires a detailed and complete global map. Traditional methods lack detailed maps of different indoor planes, and traditional reinforcement learning algorithms lack reward mechanisms for the vertical direction (z-axis), resulting in ineffective identification of staircases or elevator paths and a high failure rate for cross-floor tasks. Traditional algorithms cannot handle static and dynamic obstacles well, causing AI to easily get stuck in dynamic environments, have poor adjustment capabilities when encountering sudden events (such as being hit by a car in an urban scene), poor obstacle avoidance capabilities, and a tendency to lose direction, leading to task failure. Traditional algorithms rely on RGB image input, causing excessive network load and resource consumption, poor obstacle recognition capabilities, difficulty in learning effective strategies, and low utilization of visual information.

[0032] In view of this, some embodiments of this application provide a method for mapless navigation in an open 3D open world. This method predicts the next action of the agent by inputting image features of the current game interface and pose vector features of the agent into a fusion network. The reward value is determined by analyzing the actual position of the agent after performing the next action. This reward value can evaluate the accuracy of the next action generation, and the fusion network is optimized based on this to obtain a target fusion model. Some embodiments of this application can analyze the image features of the current screen in an open 3D open world mapless game scene in real time using the target fusion model. This enables real-time recognition of the current game scene, improves the agent's flexible adjustment capabilities, achieves accurate navigation of the agent, improves task completion efficiency and user gaming experience, and reduces reliance on static maps.

[0033] The following is in conjunction with the appendix Figure 1 The overall composition and structure of a system for mapless navigation based on an open 3D world, provided by some embodiments of this application, are illustrated by way of example.

[0034] like Figure 1As shown in the figure, some embodiments of this application provide a system diagram for mapless navigation based on an open 3D open world. This system may include a terminal 100 and a server 200. The terminal 100 can send a screenshot of the currently displayed game interface and the agent's current pose information to the server 200. The server 200 can analyze the screenshot to obtain image features; convert the pose information into pose vector features; then input the image features and pose vector features into a fusion network to be optimized, and output the agent's next action; after the agent executes the next action, a reward value is determined based on the agent's actual position, and the fusion network is optimized using the reward value to obtain a target fusion model. Subsequently, the target fusion model can provide accurate directional navigation and action guidance for the agent in an open 3D open world mapless game scene, efficiently completing game tasks.

[0035] In some embodiments of this application, the terminal 100 can be a mobile terminal or a non-portable computer terminal, and the embodiments of this application are not specifically limited here.

[0036] The following is in conjunction with the appendix Figure 2 The present application provides an exemplary embodiment of the implementation process of mapless navigation based on an open 3D world, executed by a server 200.

[0037] Please see the appendix Figure 2 , Figure 2 A flowchart of a method for mapless navigation based on an open 3D open world is provided for some embodiments of this application. The method for mapless navigation based on an open 3D open world includes:

[0038] S210, the image features of the game interface currently displayed on the screen and the pose vector features of the agent are input into the fusion network to predict the next action of the agent; wherein, the image features include: object information and relative distance between objects; the next action includes: the action type and movement direction of the agent.

[0039] For example, in some embodiments of this application, the image features of the currently displayed game interface and the pose vector features of the agent are concatenated, and feature fusion and dimensionality compression are performed through a fusion network composed of 256 fully connected layers to output the agent's next action. The image features may include information such as different objects (e.g., trees, houses, vehicles, etc.) within the game interface on the current screen and the relative spatial distances between objects. The next action includes the action type (e.g., walk, run, crouch, jump, etc.) and the direction of movement (e.g., forward, backward, left, right, left front, left rear, right front, right rear, etc.). The pose vector features may contain 11 dimensions of data, including the agent's own position, target position, orientation angle, distance, and obstacle markers. This pose vector feature structure can be simplified (e.g., removing the orientation angle) or expanded (adding speed, acceleration, etc.) according to the actual platform or task. The image features and the content of the next action can be flexibly set, and the embodiments of this application are not limited to this. The fusion network can be an LSTM (Long-Short Term Memory) fusion model.

[0040] In some embodiments of this application, before executing S210, the method for mapless navigation based on an open 3D world includes: S201, acquiring image information of the game interface currently displayed on the screen, and acquiring pose information of the agent in the current screen; analyzing the image information to determine image features in the image information; and generating the pose vector feature corresponding to the pose information.

[0041] For example, in some embodiments of this application, image information displayed on the screen is obtained by taking a screenshot of the content currently displayed on the screen. A multimodal parallel processing architecture is used to analyze and process the image information to obtain image features. The pose information is composed of the agent's own coordinates, the target point coordinates, distance, azimuth angle, and whether there are obstacles in front. Then, a fully connected layer network (which can be called the vector branch) is used to convert the pose information into pose vector features; the pose information can be normalized. The fully connected layer network maps the multidimensional state vector (i.e., pose information) to a high-dimensional feature space, and uses the ReLU activation function to enhance the nonlinear expressive power. The multimodal parallel processing architecture (which can be called the image processing branch) may include a pre-trained instance segmentation network and a monocular deep network. It should be noted that the image processing branch and the vector branch can be Transformer structures. The convolutional structure of the image processing branch (Conv2d 8→16→32→64) can be replaced with MobileNet or ResNet variants to adapt to different computing power platforms. The vector branch can be composed of different MLP depths or structures, as long as it can complete the embedding mapping and feature representation of the state vector.

[0042] In some embodiments of this application, the image features include: an instance segmentation map and a relative depth map; 201 may include: inputting the image information into a pre-trained instance segmentation network to obtain the instance segmentation map; wherein the instance segmentation map represents the object information of various objects contained in the image information; inputting the image information into a pre-trained monocular depth network to obtain the relative depth map; wherein the relative depth map represents the relative distance between various objects.

[0043] For example, in some embodiments of this application, the instance segmentation network can adopt a YOLO11 real-time inference + channel separation structure. Image information obtained from screenshots is input into the trained instance segmentation network and monocular depth estimation network (i.e., monocular depth network) respectively to achieve object segmentation and relative distance determination in the image information. For example, the instance segmentation network can process obstacle information, feasible region information, and other relevant information contained in the image information. For instance, a box is an obstacle, a region that cannot be passed through, while a road is a region that can be walked through, enabling the agent to make decisions to avoid obstacles and take feasible paths. The network in the multimodal parallel processing architecture can adopt a four-level progressive convolutional layer structure, with each convolutional layer followed by a LayerNorm normalization layer and a ReLU activation function. The convolution stride is designed to be 2, progressively compressing the feature map size output by each convolutional layer, and finally unfolding it into a multidimensional feature vector (as a specific example of image features) through a Flatten layer.

[0044] The instance segmentation network can also adopt a lightweight solution of YOLO+ category mask post-processing; the monocular depth estimation network can be replaced by binocular vision, LiDAR depth map (as in actual robot deployment) or other depth reconstruction techniques.

[0045] S220, by analyzing the actual position of the agent after performing the next action, a reward value is determined; wherein, the reward value includes a positive reward value and a negative reward value; the positive reward value indicates that the agent is gradually approaching the task target point, and the negative reward value indicates that the agent is gradually moving away from the task target point or gradually approaching the target obstacle.

[0046] For example, in some embodiments of this application, after the fusion network outputs the agent's next action, the actual position of the agent after executing the next action can be evaluated to determine the effectiveness of the next action and provide a corresponding reward value. That is, based on the currently selected next action, a series of subsequent rewards are determined, forming a total reward that is fed back to the fusion network. This can be understood as evaluating the rationality of the action decision or the path taken through the reward value. For example, if the reward value is positive, it indicates that the action decision is correct and the agent is gradually moving closer to the task objective; if the reward value is negative, it indicates that the action decision is incorrect, and the agent may be gradually moving away from the task objective or approaching obstacles, hindering the completion of the game task.

[0047] It should be noted that when analyzing image features and pose vector features, the fusion network can calculate the action space probability values ​​of multiple actions that the agent can perform, and select the action corresponding to the maximum value in the action space probability value (i.e., the next action) for output, thus realizing action space decision-making.

[0048] In some embodiments of this application, the reward values ​​may include: target point achievement reward, dynamic distance reward, and obstacle reward, etc., and different types of rewards can correspond to different reward functions. The three types of rewards are assigned a reward priority order, for example, target point achievement reward > obstacle reward > dynamic distance reward. In practical applications, at least one of the three rewards can be selected to optimize the fusion model. In this case, a multi-stage reward function can be set, which is not specifically limited in this embodiment.

[0049] In some embodiments of this application, S220 may include: when it is determined that the relative distance between the actual location and the task target point is less than a distance threshold, then triggering the generation of the positive reward value.

[0050] For example, in some embodiments of this application, the reward for achieving a target point is determined by comparing the Euclidean distance (as a specific example of relative distance) between the agent's actual position and the target point with a distance threshold. The target point can be the endpoint or intermediate point of the entire task (i.e., in long-distance pathfinding, some points are set between the start and end points of the path to facilitate reaching the endpoint; these points are intermediate points). When the Euclidean distance between the agent and the target point is less than the distance threshold, it indicates that the agent is gradually approaching the target point, and a one-time high reward is immediately triggered (as a specific example of a positive reward value); otherwise, a negative reward value or no reward is given.

[0051] The specific values ​​of high or low rewards can be preset or determined through a simulated functional relationship between distance and reward value; however, this application is not limited to these methods. Furthermore, the distance threshold can be flexibly set according to the actual game task.

[0052] In real-world game scenarios, agents may encounter situations such as climbing slopes, walking up stairs, or taking elevators while performing tasks. For such movements with height differences (if the ground in the game is set as the XY plane in the coordinate axis, then the height belongs to the Z axis, representing the distance from the ground), this application proposes a dynamic distance reward that evaluates changes in both the XY plane (i.e., the horizontal direction) and the Z axis direction (i.e., the vertical direction). That is, positive or negative rewards (i.e., positive reward values ​​or negative reward values) are given based on whether the agent is close to the task target point.

[0053] Regarding dynamic distance rewards, in some embodiments of this application, when there is a height difference between the task target point and the current position of the agent, S220 may further include:

[0054] S221, if the horizontal distance between the agent's horizontal position and the task target point is not less than a distance threshold, then the reward value is calculated according to a preset rule.

[0055] For example, in some embodiments of this application, when the agent's actual position in the horizontal plane on the XY plane is greater than or equal to a distance threshold, the reward value is calculated according to the three-dimensional Euclidean distance (as a specific example of a preset rule).

[0056] S222, if the horizontal distance between the horizontal position and the task target point is less than a distance threshold and the vertical distance between the agent and the task target point is greater than a height threshold, then the reward value is generated in the vertical direction.

[0057] For example, in some embodiments of this application, when the agent moves to its actual position in both the XY plane and the Z-axis after performing the next action, if the horizontal distance between the agent's ground position and the horizontal point on the ground (the vertical projection of the task target point) is less than a distance threshold (e.g., 20 meters) in the XY plane, and the Z-axis height difference (i.e., vertical distance) between the agent's height and the height of the task target point is greater than a height threshold (e.g., 1 meter), then double the reward is given for the change in the Z-axis direction (as a specific example of the reward value). By setting a specific Z-axis direction reward, the agent is ensured to have a high degree of adaptability in complex terrains (such as buildings and platforms).

[0058] S223, if the horizontal distance between the horizontal position and the task target point is less than a distance threshold and the vertical distance between the vertical height and the task target point is not greater than a height threshold, then the reward value is generated in the horizontal direction.

[0059] For example, in some embodiments of this application, when the agent moves to its actual position on both the XY plane and the Z axis after performing the next action, if the horizontal distance between the agent's ground position and the horizontal point on the ground where the vertical projection of the task target point is perpendicular to the ground is less than a distance threshold on the XY plane, and the Z-axis height difference between the agent's height on the Z axis and the height of the task target point is less than or equal to a height threshold, then only the changes on the XY plane are rewarded.

[0060] In addition, if there is an elevation difference between the task target point and the actual position of the current agent, and no effective Z-axis reward is obtained within a certain period of time, a time decay penalty mechanism is activated (i.e., a negative reward value is given).

[0061] The reward value generated in the vertical direction is within the set reward range.

[0062] For example, to avoid reward oscillations, a change filter is set for the Z-axis reward (e.g., ignoring values ​​within 0.1 meters), and the maximum / minimum reward is limited (as a specific example of the reward range).

[0063] It should be understood that the distance threshold, height threshold, and reward range can be flexibly adjusted according to the actual game scenario, and the embodiments of this application do not impose specific limitations here.

[0064] Regarding obstacle rewards, in some embodiments of this application, if it is confirmed that the distance between the actual position and the target obstacle is less than the distance between the agent's current position and the target obstacle, then the negative reward value is generated; the negative reward value indicates that the agent is gradually approaching the target obstacle.

[0065] For example, in some embodiments of this application, before the agent performs the next action, it first determines the distance between its current position and the target obstacle. After the agent performs the next action, it determines the actual position and the distance between the target obstacle. If the distance after the action is smaller than before, it indicates that the agent is approaching the target obstacle and cannot achieve intelligent obstacle avoidance. In this case, a negative reward value is given.

[0066] Alternatively, pixel-level detection can be performed on the bottom center region of the instance segmentation map output by the instance segmentation network. The segmented pixel values ​​are compared with the pre-configured obstacle colors to see if they are within the pixel tolerance range (e.g., ±10). If they are, it indicates that the obstacle in the image information has been accurately segmented, and a positive reward value can be given; otherwise, it indicates that the obstacle has not been accurately segmented, and a negative penalty, i.e., a negative reward value, can be given. The ROI region size can be adaptively adjusted according to the resolution (e.g., 480p or 240p). ROI bottom obstacle recognition closely resembles the human visual field model, which can significantly improve navigation robustness.

[0067] In addition, this application can set hyperparameters to control the range of change in reward values ​​to prevent outliers; at the same time, the range of change in reward values ​​is controlled within an order of magnitude to prevent the strategy from being dominated by a single factor.

[0068] S230, adjust the network parameters of the fusion network based on the reward value to obtain the target fusion model; wherein, the target fusion model is used to provide the agent with action and direction navigation when the agent performs the game task.

[0069] For example, in some embodiments of this application, the reward values ​​obtained in the above embodiments are used to optimize the parameters of the fusion network to obtain a target fusion model that can accurately provide action and direction navigation for the agent.

[0070] When performing game tasks, by inputting the image features corresponding to the image information displayed on the current game interface and the pose vector features of the agent into the target fusion model, the next action and direction to be executed by the agent can be output, thereby achieving precise navigation of the agent and improving the efficiency of completing game tasks.

[0071] The following is in conjunction with the appendix Figure 3 The present application provides an exemplary description of the specific process of mapless navigation based on an open 3D world, using some embodiments of this application.

[0072] Please see the appendix Figure 3 , Figure 3 A flowchart of a method for mapless navigation based on an open 3D open world is provided for some embodiments of this application.

[0073] The above process is illustrated below by example.

[0074] S310 captures the image information of the game interface currently displayed on the screen.

[0075] S320 inputs image information into the instance segmentation network and the monocular depth network respectively, and outputs image features.

[0076] S330, acquires the pose vector features corresponding to the pose information of the agent.

[0077] S340 inputs image features and pose vector features into a fusion network to predict the agent's next action.

[0078] S350 determines the reward value by analyzing the actual position of the agent after it performs the next action.

[0079] S360 adjusts the network parameters of the fusion network based on the reward value to obtain the target fusion model.

[0080] It should be noted that S320 and S330 can be executed simultaneously, or S330 can be executed first and then S320. This application does not make specific limitations here. The specific implementation process of S310 to S360 can be referred to the method embodiments provided above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0081] As can be seen from the above embodiments of this application, this application has the following advantages compared with traditional methods:

[0082] 1) Reduce reliance on static maps and improve adaptability to dynamic environments;

[0083] Traditional methods rely on NavMesh or global map planning, which cannot cope with real-time obstacle changes, resulting in poor system robustness. This invention replaces the static navigation map with a dynamic visual input mechanism based on instance segmentation maps and depth estimation maps; and through semantic detection of the bottom ROI region, it accurately detects and penalizes potential obstacle collision areas at each step. This design combines a dynamic distance reward function with an obstacle penalty mechanism (i.e., obstacle reward), enabling the system to proactively adapt to dynamic environments. Experimental results show that even when encountering high-speed vehicle interference in urban roads, the agent can still maintain its navigation direction, significantly outperforming traditional static map navigation strategies.

[0084] 2) Cross-floor three-dimensional spatial navigation capabilities are significantly enhanced;

[0085] Traditional reinforcement learning strategies lack explicit Z-axis rewards, leading to a loss of exploration ability for upper and lower floors in multi-story structures. This invention addresses this by employing a Z-axis-specific reward mechanism and a distance-related weighting strategy to effectively guide paths with elevation differences. The system accurately identifies pathways with elevation differences, such as stairs and elevators, and automatically triggers a penalty mechanism when the Z-axis reward interval exceeds 30 seconds, preventing the strategy from remaining on the same plane for extended periods. In actual testing, the agent achieved a completion rate of up to 75% in indoor building navigation tasks requiring multiple floor crossings, significantly outperforming traditional algorithms lacking elevation awareness mechanisms.

[0086] 3) Improve the stability and generalization ability of navigation strategies.

[0087] By introducing a multimodal network structure, this invention can fully utilize information from the image processing branch and the vector branch: the image processing branch uses a four-level progressive convolutional network to extract multi-scale semantic features, while the vector branch extracts environmental information such as the current state, orientation, and obstacle placement. The fused features are modeled in the time dimension using LSTM (as a specific example of a sub-model within the fusion network), and the output is jointly decided by the policy branch and the value branch. This fusion network optimizes policy updates using the PPO algorithm, effectively avoiding policy oscillation problems. Comparative experiments show that this network structure exhibits high robustness and policy stability in multiple different terrain tasks, good transferability, and faster training convergence.

[0088] 4) Integrating relative depth maps and instance segmentation maps overcomes training limitations;

[0089] By combining relative depth maps and instance segmentation maps as input, and fully integrating spatial geometric and semantic structural information, more accurate navigable area identification can be achieved. Furthermore, multi-scale fusion is performed through a more stable image feature extraction network, effectively enhancing policy awareness and generalization capabilities across floors, complex terrains, and occluded environments. The fusion of relative depth maps and instance segmentation maps is a key supporting technology for achieving highly robust 3D navigation. Through dual-modal input, the model not only achieves geometric and semantic complementarity but also effectively overcomes the cognitive limitations and training bottlenecks caused by a single perception source, significantly improving the overall task completion rate and stability.

[0090] In summary, this application effectively addresses the key technical shortcomings of existing technologies in dynamic obstacle avoidance, complex path planning, and 3D structure understanding through systematic technical improvements in reward mechanism design, perception structure optimization, and spatiotemporal information fusion. It achieves repeatable, high-success-rate, and highly robust navigation control tasks in complex environment navigation scenarios, demonstrating significant technological advancement and application promotion value.

[0091] Please refer to Figure 4 , Figure 4 The diagram illustrates the composition of an apparatus for mapless navigation based on an open 3D world, provided in some embodiments of this application. It should be understood that this apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0092] Figure 4A device for open 3D open-world mapless navigation includes at least one software functional module that can be stored in a memory or embedded in the device in the form of software or firmware. The device includes: a fusion module 410, used to input image features of the currently displayed game interface and pose vector features of the agent into a fusion network to predict the agent's next action; wherein the image features include: object information and relative distances between objects; the next action includes: the agent's action type and direction of movement; a reward module 420, used to analyze the agent's actual position after performing the next action to determine a reward value; wherein the reward value includes positive and negative reward values; the positive reward value indicates that the agent is gradually approaching the task target point, and the negative reward value indicates that the agent is gradually moving away from the task target point or gradually approaching a target obstacle; and an optimization module 430, used to adjust the network parameters of the fusion network based on the reward value to obtain a target fusion model; wherein the target fusion model is used to provide action and direction navigation for the agent when the agent performs a game task.

[0093] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0094] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0095] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0096] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement the methods of any of the above embodiments.

[0097] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0098] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0099] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0101] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for mapless navigation based on an open 3D world, characterized in that, include: The image features of the game interface currently displayed on the screen and the pose vector features of the agent are input into a fusion network to predict the agent's next action; wherein, the image features include: object information and relative distances between objects; the next action includes: the action type and movement direction of the agent; By analyzing the actual position of the agent after executing the next action, a reward value is determined; wherein, the reward value includes a positive reward value and a negative reward value; the positive reward value indicates that the agent is gradually moving closer to the task target point, and the negative reward value indicates that the agent is gradually moving away from the task target point or gradually moving closer to the target obstacle; The network parameters of the fusion network are adjusted based on the reward value to obtain a target fusion model; wherein, the target fusion model is used to provide the agent with action and direction navigation when the agent performs game tasks; the image features include: instance segmentation map and relative depth map; Before inputting the image features of the game interface currently displayed on the screen and the pose vector features of the agent into the fusion network, the method further includes: Obtain image information of the game interface currently displayed on the screen, and obtain pose information of the intelligent agent in the current screen; The image information is input into a pre-trained instance segmentation network to obtain the instance segmentation map; wherein, the instance segmentation map represents the object information of various objects contained in the image information; the image information is input into a pre-trained monocular depth network to obtain the relative depth map; wherein, the relative depth map represents the relative distance between various objects; Generate the pose vector feature corresponding to the pose information.

2. The method as described in claim 1, characterized in that, The step of determining the reward value by analyzing the actual position of the agent after it performs the next action includes: When the relative distance between the actual location and the task target point is determined to be less than a distance threshold, the positive reward value is generated.

3. The method as described in claim 1, characterized in that, When there is a height difference between the task target point and the current position of the agent, the actual position includes the horizontal position and the vertical height from the ground; The step of determining the reward value by analyzing the actual position of the agent after it performs the next action includes: If the horizontal distance between the agent's horizontal position and the task target point is not less than a distance threshold, then the reward value is calculated according to a preset rule; If the horizontal distance between the horizontal position and the task target point is less than a distance threshold and the vertical distance between the agent and the task target point is greater than a height threshold, then the reward value is generated in the vertical direction. If the horizontal distance between the horizontal position and the task target point is less than a distance threshold and the vertical distance between the vertical position and the task target point is not greater than a height threshold, then the reward value is generated in the horizontal direction.

4. The method as described in claim 3, characterized in that, The reward value generated in the vertical direction is within the set reward range.

5. The method as described in claim 1, characterized in that, The step of determining the reward value by analyzing the actual position of the agent after it performs the next action includes: If the distance between the actual location and the target obstacle is confirmed to be less than the distance between the agent's current location and the target obstacle, then the negative reward value is generated; the negative reward value indicates that the agent is gradually approaching the target obstacle.

6. A device for map-free navigation based on an open 3D world, characterized in that, The device can perform the method as described in any one of claims 1-5, including: The fusion module is used to input the image features of the game interface currently displayed on the screen and the pose vector features of the agent into the fusion network to predict the agent's next action; wherein, the image features include: object information and relative distances between objects; the next action includes: the action type and movement direction of the agent; The reward module is used to determine a reward value by analyzing the actual position of the agent after it performs the next action; wherein the reward value includes a positive reward value and a negative reward value; the positive reward value indicates that the agent is gradually moving closer to the task target point, and the negative reward value indicates that the agent is gradually moving away from the task target point or gradually moving closer to the target obstacle; An optimization module is used to adjust the network parameters of the fusion network based on the reward value to obtain a target fusion model; wherein, the target fusion model is used to provide the agent with action and direction navigation when the agent performs game tasks.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-5.

Images