Multi-dimensional state space intelligent network optimization method

Abstract

The invention particularly relates to a method for optimizing an intelligent network with a body in a multi-dimensional state space. According to the method, a multi-dimensional state space S and a hierarchical action space A are constructed, a model based on deep reinforcement learning is used as a core scheduler, and autonomous learning and collaborative decision making of the method are realized; building a multi-dimensional space, building a hierarchical action space and a reinforcement learning collaborative optimization model are seamlessly integrated into a control system of the intelligent agent, and autonomous behavior decision-making of network awareness is achieved; the active optimization framework efficiently interacts with a robot operating system (ROS) service of the robot body, and converts an abstract optimization strategy into specific physical and communication instructions. According to the multi-dimensional state space-based intelligent network optimization method, the transformation of the intelligent agent from the traditional passive adaptation network to the active optimization network connection is realized, and the end-to-end task performance and the connection stability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to an embodied intelligent network optimization method with a multi-dimensional state space. Background Technology

[0002] In today's technological field, embodied intelligent agents, as intelligent systems with physical bodies and the ability to perceive and act, are widely used in many critical mission scenarios such as inspection, delivery, and search and rescue. However, these complex and ever-changing mission environments pose significant challenges to the operation of embodied intelligent agents. Due to their mobility, coupled with interference from obstacles in the environment and dynamically changing wireless channel conditions, the stable, high-quality wireless communication connections they rely on are frequently at risk of interruption, and network quality fluctuates frequently.

[0003] In contrast, existing network optimization methods have obvious limitations.

[0004] On the one hand, the decision-making dimensions are relatively simple, mostly focusing on operations such as link selection and power control within the communication domain. They merely treat the physical location of the embodied agent as a given input or constraint of the environment, without taking its active position movement into the scope of decision-making considerations.

[0005] On the other hand, due to the lack of coordination capabilities, traditional methods have not established an intrinsic connection between communication behaviors (such as switching links) and physical behaviors (such as moving to a location with better signal). This makes it difficult for agents to passively adapt when faced with network deterioration, and they cannot actively improve communication quality by changing their own physical state. Consequently, they cannot effectively guarantee end-to-end performance in critical tasks.

[0006] To address the aforementioned problems, this invention proposes an embodied intelligent network optimization method with a multi-dimensional state space. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a simple and efficient method for optimizing embodied intelligent networks in a multidimensional state space.

[0008] This invention is achieved through the following technical solution: A method for optimizing embodied intelligent networks in a multidimensional state space includes the following steps: Step S1: Construct a multidimensional state space Define a multidimensional state space S for an embodied intelligent agent, which consists of the physical state information S of the embodied intelligent agent. phy and network status information S net It is constructed by fusion, that is, S={S phy S net}; In physical state information S phyWithin this dimension, the core information includes the agent's current position coordinates and movement speed, which are used to characterize the agent's spatial position and movement trends. Physical state information S phy The dimensions also include task-related information, such as the distance to the task target, task progress, and energy consumption status, to assess the cost of movement and task efficiency. In network status information S net Within this dimension, core metrics are used to evaluate the real-time communication quality of all candidate links, including: The signal-to-noise ratio of each candidate link is used to measure signal strength and interference; Network latency is a measure of data transmission efficiency. Packet loss rate is used to reflect the reliability of a connection; To enhance the robustness of decision-making, network state information S net It also includes link throughput and resource load information.

[0009] In step S1, the physical state information S is normalized using the Z-Score method. phy and network status information S net Perform standardization processing; for a certain feature x i Its standardized value z i = (x i -μ) / σ; Where μ is the feature x i The average value, σ is the characteristic x i The standard deviation.

[0010] Step S2: Constructing a hierarchical action space Design a hierarchical action space A, and decompose the agent's decision-making into communication action information A. comm and physical motion information A phy This enables deep collaboration between communication and physical behaviors, allowing intelligent agents to collaboratively make decisions on "where to communicate" and "how to communicate" based on information from the multidimensional state space S. The communication action information A comm Located at the bottom level of decision-making, including: Link selection or switching is the process of choosing the optimal link from the currently available candidate links to maximize instantaneous network quality. Power control, also known as dynamic adjustment of transmit power levels, is used to balance connection quality and energy consumption and reduce interference to other users. Resource requests are requests made to the network controller for specific bandwidth or quality of service (QoS) levels. The physical action information A phyLocated at a high level of decision-making, it represents the intelligent agent actively changing its own physical state to fundamentally improve communication channel conditions. The core physical action is active position movement, that is, deciding to move to a preset or calculated target position with better signal quality in order to eliminate or reduce obstruction, reduce distance attenuation, and physically improve channel conditions. When the model determines that in communication action information A comm If network quality cannot be effectively improved, or if the sustained positive benefits of the movement over a predetermined future period outweigh the direct costs of the movement, the model will decide to execute physical action information A. phy Achieve proactive optimization; Other physical actions include: Task pause or wait means that when the network status index is lower than a custom threshold or the mobility cost is higher than a custom threshold, the task is temporarily stopped or waited for the channel conditions to recover naturally in order to save energy. Attitude adjustment refers to adjusting the orientation of the agent or the angle of the antenna to optimize directional gain.

[0011] When the model determines that in communication action information A comm If network quality cannot be effectively improved, or if the sustained positive benefits of the movement over a predetermined future period outweigh the direct costs of the movement, the model will decide to execute physical action information A. phy Achieve proactive optimization; Step S3: Reinforcement Learning Collaborative Optimization Model Using a deep reinforcement learning-based model as the core scheduler, the model achieves autonomous learning and collaborative decision-making for the optimization method of embodied agent networks. The core objective of the model is to learn an optimal policy π(s), which is a mapping from the multidimensional state space S to hierarchical actions A, thereby maximizing long-term cumulative rewards. A deep neural network is used as a function approximator for reinforcement learning. The network input layer receives a standardized multidimensional state vector S, which contains fused information of physical and network states. The network output layer corresponds to hierarchical actions A, which can simultaneously handle discrete communication actions and continuous physical actions. To effectively handle this complex continuous-discrete hybrid action space, an Actor-Critic (A2C / A3C) architecture is adopted, in which the Actor network is responsible for outputting action policies, and the Critic network is responsible for evaluating the value of the current state. The reward function R is designed as the core to guide the reinforcement learning agent in its learning process. It comprehensively considers communication performance, task efficiency, and movement costs to incentivize the model to make globally optimal collaborative decisions. This scheme designs the following comprehensive reward function: Reward function R= ×R task+β×R net -γ×R move in, Rewards for task efficiency Network quality rewards are used to incentivize high-quality communication connections, aiming to maximize throughput or minimize latency or packet loss rate. To penalize the movement, penalties are imposed on performing physical movement actions. These penalties can be set based on the distance, energy consumption, or time cost of the movement, in order to encourage agents to move only when necessary. , and To support the adjustment of weighting coefficients, used to balance the relationship between communication, tasks, and mobility; Step S4: Actively optimize the framework The system seamlessly integrates the construction of a multi-dimensional state space, a hierarchical action space, and a reinforcement learning collaborative optimization model into the control system of the embodied intelligent agent, enabling autonomous behavior decision-making based on network perception. The active optimization framework interacts efficiently with the robot's Robot Operating System (ROS) service, transforming abstract optimization strategies into specific physical and communication instructions. The active optimization framework operates in a periodic decision-making loop to ensure that the embodied agent can respond to environmental changes in real time; the specific steps are as follows: Step S01: Multidimensional state information acquisition and construction of multidimensional state vector S Physical state acquisition: The reinforcement learning scheduler obtains the current physical state information S by calling or subscribing to the robot's operating system ROS perception service. phy This includes location coordinates and movement speed; simultaneously, it acquires related information provided by the task execution module, including the distance to the task target; Network Status Acquisition: The scheduler obtains real-time network status information (S) for all candidate links by calling the robot's ROS communication service interface. net These include signal-to-noise ratio, network latency, and packet loss rate; State vector construction and standardization: This involves constructing and standardizing the acquired physical state information S... phy and real-time network status information S net The data are fused into a multidimensional state vector S; this vector is then Z-score normalized to ensure weight balance across different dimensions of the data, and the normalized vector is then input into the reinforcement learning model. Step S02: Hierarchical Decision Making A through Reinforcement Learning Policy reasoning: Input the standardized multidimensional state vector S into the reinforcement learning co-optimization model; Action Output: The reinforcement learning collaborative optimization model outputs hierarchical actions A = (A...) based on the current optimal policy π(s). comm A phy ); where A comm For communication action information, A phy Information about physical actions; Step S03: Calling and executing ROS services for hierarchical motion robots The reinforcement learning scheduler calls the corresponding Robot Operating System (ROS) service based on the hierarchical action A output, translating the decision into the robot's actual behavior: Communication action execution: If communication action information A comm Including link switching or power adjustment commands, the reinforcement learning scheduler will call the robot's operating system ROS network control service to send configuration update commands to the underlying communication module; Physical action execution: If physical action information A phy For proactive positional movement, the reinforcement learning scheduler will invoke the robot's Robot Operating System (ROS) motion control service, sending the calculated target movement coordinates as the navigation target to the robot's navigation system; the motion control service will then plan the motion trajectory and drive the robot to perform the movement. Step S04: Reward Calculation and Model Optimization Reward Calculation: The reinforcement learning scheduler calculates the reward value of the action in real time based on the new multidimensional state space S' after the action is executed and the preset comprehensive reward function R. Policy optimization: The experience tuple (S, A, R, S') is stored in the experience replay buffer; during training, the reinforcement learning co-optimization model periodically samples data from the buffer and updates the parameters of the deep neural network to achieve continuous policy optimization.

[0012] A multi-dimensional state-space embodied intelligent network optimization system is provided to implement the above-mentioned method. It seamlessly integrates the construction of a multi-dimensional state space, a hierarchical action space, and a reinforcement learning collaborative optimization model into the control system of the embodied intelligent agent. Interaction is achieved through a specific robot operating system (ROS) interface, transforming abstract optimization strategies into concrete physical and communication commands. Specifically, it includes the following modules: Multidimensional State Awareness Module: Running as a ROS Node in the robot operating system, it is responsible for collecting and processing the physical state information S of the embodied intelligent agent. phy and network status information S net And construct a multidimensional state vector S; Hierarchical Action Module: Running as a Robot Operating System (ROS) Node, it is responsible for outputting control commands to the embodied agent, including communication action information A. comm and physical motion information Aphy ; Reinforcement learning scheduler: It runs as an independent robot operating system node (ROS Node), runs the trained reinforcement learning model, performs policy reasoning, and outputs hierarchical actions A. During training, the reinforcement learning co-optimization model periodically samples data from the buffer and updates the parameters of the deep neural network to achieve continuous policy optimization.

[0013] A multi-dimensional state-space embodied intelligent network optimization computing device, comprising: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0014] A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an embodied intelligent network optimization computing device of a multidimensional state space, cause the embodied intelligent network optimization computing device of the multidimensional state space to perform any of the methods described above.

[0015] The beneficial effects of this invention are: the embodied intelligent network optimization method in multi-dimensional state space realizes the transformation of embodied intelligent agents from passively adapting to network changes to actively optimizing network connections, which significantly improves end-to-end task performance and connection stability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Appendix Figure 1 This is a schematic diagram of the embodied intelligent network optimization method for multidimensional state space according to the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below in conjunction with the embodiments of this invention. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0019] This method for optimizing embodied intelligent networks in a multidimensional state space includes the following steps: Step S1: Construct a multidimensional state space The concept of a multi-dimensional state space S for embodied agents aims to address the issue of single-dimensional decision-making in existing methods and achieve deep collaboration between communication and physical behavior. This state space consists of the physical state information S of the embodied agent. phy and network status information S net It is constructed by fusion, that is, S={S phy S net This multidimensional state space provides a comprehensive basis for reinforcement learning agents to make decisions, enabling them to understand the coupling relationship between the current network quality and their own physical location, thus laying the foundation for subsequent proactive optimization decisions.

[0020] In physical state information S phy Within this dimension, core information includes the current position coordinates of the agent (e.g., a humanoid robot). ) and movement speed (linear velocity and angular velocity), used to characterize the spatial position and motion trend of the intelligent agent; Physical state information S phy The dimensions also include task-related information, such as the distance to the task target, task progress, and energy consumption status, to assess the cost of movement and task efficiency. In network status information S net Within this dimension, core metrics are used to evaluate the real-time communication quality of all candidate links, including: The signal-to-noise ratio of each candidate link is used to measure signal strength and interference; Network latency is a measure of data transmission efficiency. Packet loss rate is used to reflect the reliability of a connection; To enhance the robustness of decision-making, network state information S net It also includes link throughput and resource load information.

[0021] To ensure that all dimensions of data maintain consistency in magnitude and prevent the model from being dominated by a particular metric, these multidimensional metrics must be standardized before being input into the reinforcement learning model. Z-Score standardization is particularly suitable for metrics with unknown distributions or potential outliers (such as network latency), as it can transform the original data into a distribution with a mean of 0 and a standard deviation of 1, thereby achieving a balanced weight across all features.

[0022] In step S1, the physical state information S is processed using the Z-Score standardization method (or standardization). phy and network status information S net Perform standardization processing; for a certain feature xi Its standardized value z i = (x i -μ) / σ; Where μ is the feature x i The average value, σ is the characteristic x i The standard deviation.

[0023] In this way, even state indicators with different dimensions and huge numerical differences (such as location coordinates and network latency) can be effectively integrated into the state space. In this context, it supports stable training and efficient convergence of reinforcement learning models.

[0024] Step S2: Constructing a hierarchical action space Design a hierarchical action space A, and decompose the agent's decision-making into communication action information A. comm and physical motion information A phy This enables deep collaboration between communication and physical behaviors, allowing intelligent agents to collaboratively make decisions on "where to communicate" and "how to communicate" based on information from the multidimensional state space S. The communication action information A comm Located at the bottom level of decision-making, including: Link selection or switching is the process of choosing the optimal link from the currently available candidate links to maximize instantaneous network quality (such as throughput or signal-to-noise ratio). Power control, also known as dynamic adjustment of transmit power levels, is used to balance connection quality and energy consumption and reduce interference to other users. Resource requests are requests made to the network controller for specific bandwidth or quality of service (QoS) levels. The physical action information A phy Located at a high level of decision-making, it represents the intelligent agent actively changing its own physical state to fundamentally improve communication channel conditions. The core physical action is active position movement, that is, deciding to move to a preset or calculated target position with better signal quality (such as moving towards the access point AP or avoiding obstacles) to eliminate or reduce obstruction, reduce distance attenuation, and physically improve channel conditions. When the model determines that in communication action information A comm If network quality cannot be effectively improved, or if the sustained positive benefits of the movement over a predetermined future period outweigh the direct costs of the movement, the model will decide to execute physical action information A. phy Achieve proactive optimization; Other physical actions include: Task pause or wait means that when the network status index is lower than a custom threshold or the mobility cost is higher than a custom threshold, the task is temporarily stopped or waited for the channel conditions to recover naturally in order to save energy. Attitude adjustment refers to adjusting the orientation of the agent or the angle of the antenna to optimize directional gain.

[0025] When the model determines that in communication action information A comm If network quality cannot be effectively improved, or if the sustained positive benefits of the movement over a predetermined future period outweigh the direct costs of the movement, the model will decide to execute physical action information A. phy Achieve proactive optimization; When the model determines the communication action information If network quality cannot be effectively improved, or if the sustained positive benefits of the movement over a predetermined future period outweigh the direct costs of performing the movement, the model will decide to execute the physical action information. Achieve proactive optimization; The hierarchical coordination and output of actions are achieved through a reinforcement learning model; at each time step, the model outputs a hierarchical action. The model comprehensively considers the multidimensional state space. (Especially the coupling relationship between location and network quality) determines the priority and combination of two actions, thereby enabling embodied agents to transcend traditional link scheduling and truly realize the transformation from passive adaptation to active optimization, that is, not only can they choose the best "channel", but also the best "communication location".

[0026] Step S3: Reinforcement Learning Collaborative Optimization Model Using a Deep Reinforcement Learning (DRL)-based model as the core scheduler, the model achieves autonomous learning and collaborative decision-making for the optimization method of embodied agent networks. The core objective of the model is to learn an optimal policy π(s), which is a mapping from the multidimensional state space S to hierarchical actions A, thereby maximizing long-term cumulative rewards. A deep neural network is used as a function approximator for reinforcement learning. The network input layer receives a standardized multidimensional state vector S, which contains fused information of physical and network states. The network output layer corresponds to hierarchical actions A, which can simultaneously handle discrete communication actions (e.g., link ID selection) and continuous physical actions (e.g., moving target coordinates). To effectively handle this complex continuous-discrete hybrid action space, an Actor-Critic (A2C / A3C) architecture is adopted, in which the Actor network is responsible for outputting action policies, and the Critic network is responsible for evaluating the value of the current state. The reward function R is designed as the core to guide the reinforcement learning agent in its learning process. It comprehensively considers communication performance, task efficiency, and movement costs to incentivize the model to make globally optimal collaborative decisions. This scheme designs the following comprehensive reward function: Reward function R= ×R task +β×R net -γ×R move in, Rewards for task efficiency Network quality rewards are used to incentivize high-quality communication connections, aiming to maximize throughput or minimize latency or packet loss rate. To penalize the movement, penalties are imposed on performing physical movement actions. These penalties can be set based on the distance, energy consumption, or time cost of the movement, in order to encourage agents to move only when necessary. , and To support the adjustment of weighting coefficients, used to balance the relationship between communication, tasks, and mobility; Step S4: Actively optimize the framework The system seamlessly integrates the construction of a multi-dimensional state space, a hierarchical action space, and a reinforcement learning collaborative optimization model into the control system of an embodied intelligent agent (such as a humanoid robot) to achieve autonomous behavior decision-making based on network perception. The active optimization framework interacts efficiently with the robot's Robot Operating System (ROS) service to transform abstract optimization strategies into specific physical and communication instructions. The active optimization framework operates in a periodic decision-making loop to ensure that the embodied agent can respond to environmental changes in real time; the specific steps are as follows: Step S01: Multidimensional state information acquisition and construction of multidimensional state vector S Physical state acquisition: The reinforcement learning scheduler obtains the current physical state information S by calling or subscribing to the robot's operating system ROS perception services (such as topics related to position, attitude, and velocity information). phy This includes location coordinates and movement speed; simultaneously, it acquires related information provided by the task execution module, including the distance to the task target; Network Status Acquisition: The scheduler obtains real-time network status information (S) for all candidate links by calling the robot's ROS communication service interface. net These include signal-to-noise ratio, network latency, and packet loss rate; State vector construction and standardization: This involves constructing and standardizing the acquired physical state information S... phy and real-time network status information S netThe data are fused into a multidimensional state vector S; this vector is then Z-score normalized to ensure weight balance across different dimensions of the data, and the normalized vector is then input into the reinforcement learning model. Step S02: Hierarchical Decision Making A through Reinforcement Learning Policy reasoning: Input the standardized multidimensional state vector S into the reinforcement learning co-optimization model; Action Output: The reinforcement learning collaborative optimization model outputs hierarchical actions A = (A...) based on the current optimal policy π(s). comm A phy ); where A comm For communication action information (such as link switching ID), A phy Information about physical actions; Step S03: Calling and executing ROS services for hierarchical motion robots The reinforcement learning scheduler calls the corresponding Robot Operating System (ROS) service based on the hierarchical action A output, translating the decision into the robot's actual behavior: Communication action execution: If communication action information A comm Including link switching or power adjustment commands, the reinforcement learning scheduler will call the robot's ROS network control service (e.g., / net / link_switch Action) to send configuration update commands to the underlying communication module; Physical action execution: If physical action information A phy For proactive positional movement, the reinforcement learning scheduler will invoke the robot's Robot Operating System (ROS) motion control service (e.g., / robot / move_base Action), send the calculated target movement coordinates as the navigation target to the robot's navigation system, plan the motion trajectory through the motion control service, and drive the robot to perform the movement. Step S04: Reward Calculation and Model Optimization Reward Calculation: The reinforcement learning scheduler calculates the reward value of the action in real time based on the new multidimensional state space S' after the action is executed and the preset comprehensive reward function R. Policy optimization: The experience tuple (S, A, R, S') is stored in the experience replay buffer; during training, the reinforcement learning co-optimization model periodically samples data from the buffer and updates the parameters of the deep neural network to achieve continuous policy optimization.

[0027] This multi-dimensional state-space embodied intelligent network optimization system is used to implement the above methods. It seamlessly integrates the construction of a multi-dimensional state space, the construction of a hierarchical action space, and a reinforcement learning collaborative optimization model into the control system of an embodied intelligent agent (e.g., a humanoid robot). It interacts through a specific robot operating system ROS interface, transforming abstract optimization strategies into concrete physical and communication commands. Specifically, it includes the following modules: Multidimensional State Awareness Module: Running as a Robot Operating System (ROS) Node, it is responsible for collecting and processing the physical state information S of the embodied intelligent agent (e.g., a humanoid robot). phy and network status information S net And construct a multidimensional state vector S; Hierarchical Action Module: Running as a Robot Operating System (ROS) Node, it is responsible for outputting control commands to the embodied agent, including communication action information A. comm and physical motion information A phy ; Reinforcement learning scheduler: It runs as an independent robot operating system node (ROS Node), runs the trained reinforcement learning model, performs policy reasoning, and outputs hierarchical actions A. During training, the reinforcement learning co-optimization model periodically samples data from the buffer and updates the parameters of the deep neural network to achieve continuous policy optimization.

[0028] This embodied intelligent network optimization computing device for the multidimensional state space includes: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0029] The computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by an embodied intelligent network optimization computing device of a multidimensional state space, cause the embodied intelligent network optimization computing device of the multidimensional state space to perform any of the methods described above.

[0030] This embodied intelligent network optimization method in a multi-dimensional state space constructs a multi-dimensional state space by integrating the agent's position information and network quality indicators, and designs hierarchical actions from link selection to active position movement. This enables the embodied intelligent agent to transform from passively adapting to network changes to actively optimizing network connections, thereby significantly improving end-to-end task performance and connection stability.

[0031] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing embodied intelligent networks in a multi-dimensional state space, characterized in that: Includes the following steps: Step S1: Construct a multidimensional state space Define a multidimensional state space S for an embodied intelligent agent, which consists of the physical state information S of the embodied intelligent agent. phy and network status information S net It is constructed by fusion, that is, S={S phy S net }; In physical state information S phy Within this dimension, the core information includes the agent's current position coordinates and movement speed, which are used to characterize the agent's spatial position and movement trends. In network status information S net Within this dimension, core metrics are used to evaluate the real-time communication quality of all candidate links, including: The signal-to-noise ratio of each candidate link is used to measure signal strength and interference; Network latency serves as a measure of data transmission efficiency; Packet loss rate is used to reflect the reliability of a connection; Step S2: Constructing a hierarchical action space Design a hierarchical action space A, and decompose the agent's decision-making into communication action information A. comm and physical motion information A phy This enables deep collaboration between communication and physical behaviors, allowing intelligent agents to collaboratively make decisions on "where to communicate" and "how to communicate" based on information from the multidimensional state space S. The communication action information A comm Located at the bottom level of decision-making, including: Link selection or switching is the process of choosing the optimal link from the currently available candidate links to maximize instantaneous network quality. Power control, also known as dynamic adjustment of transmit power levels, is used to balance connection quality and energy consumption and reduce interference to other users. Resource requests are requests made to the network controller for specific bandwidth or quality of service levels. The physical action information A phy Located at a high level of decision-making, it represents the intelligent agent actively changing its own physical state to fundamentally improve communication channel conditions. The core physical action is active position movement, that is, deciding to move to a preset or calculated target position with better signal quality in order to eliminate or reduce obstruction, reduce distance attenuation, and physically improve channel conditions. When the model determines that in communication action information A comm If network quality cannot be effectively improved, or if the sustained positive benefits of the movement over a predetermined future period outweigh the direct costs of the movement, the model will decide to execute physical action information A. phy Achieve proactive optimization; Step S3: Reinforcement Learning Collaborative Optimization Model Using a deep reinforcement learning-based model as the core scheduler, the model achieves autonomous learning and collaborative decision-making for the optimization method of embodied agent networks. The core objective of the model is to learn an optimal policy π(s), which is a mapping from the multidimensional state space S to hierarchical actions A, thereby maximizing long-term cumulative rewards. A deep neural network is used as a function approximator for reinforcement learning; the network input layer receives a standardized multidimensional state vector S, which contains fused information of physical state and network state; the network output layer corresponds to the hierarchical action A, which can simultaneously process discrete communication actions and continuous physical actions. Design a reward function R as the core to guide the reinforcement learning agent in learning, taking into account communication performance, task efficiency and movement cost, so as to incentivize the model to make a globally optimal collaborative decision; Step S4: Actively optimize the framework The system seamlessly integrates the construction of a multi-dimensional state space, a hierarchical action space, and a reinforcement learning collaborative optimization model into the control system of the embodied intelligent agent, enabling autonomous behavior decision-making based on network perception. The active optimization framework interacts efficiently with the robot's operating system ROS service to transform abstract optimization strategies into specific physical and communication instructions.

2. The method for optimizing embodied intelligent networks in a multidimensional state space according to claim 1, characterized in that: Physical state information S phy The dimensions also include task-related information, such as the distance to the task target, task progress, and energy consumption status, to assess the cost of movement and task efficiency. To enhance the robustness of decision-making, network state information S net It also includes link throughput and resource load information.

3. The method for optimizing embodied intelligent networks in a multidimensional state space according to claim 1, characterized in that: In step S1, the physical state information S is normalized using the Z-Score method. phy and network status information S net Perform standardization processing; for a certain feature x i Its standardized value z i = (x i -μ) / σ; Where μ is the feature x i The average value, σ is the characteristic x i The standard deviation.

4. The method for optimizing embodied intelligent networks in a multidimensional state space according to claim 1, characterized in that: The physical action information A phy Also includes: Task pause or wait means that when the network status index is lower than a custom threshold or the mobility cost is higher than a custom threshold, the task is temporarily stopped or waited for the channel conditions to recover naturally in order to save energy. Attitude adjustment refers to adjusting the orientation of the agent or the angle of the antenna to optimize directional gain.

5. The method for optimizing embodied intelligent networks in a multidimensional state space according to claim 1, characterized in that: To effectively handle the continuous-discrete mixed action space, an Actor-Critic (A2C / A3C) architecture is adopted, in which the Actor network is responsible for outputting the action policy, and the Critic network is responsible for evaluating the value of the current state.

6. The method for optimizing embodied intelligent networks in a multidimensional state space according to claim 3, characterized in that: Reward function R= ×R task +β×R net -γ×R move Among them, R task As a reward for task efficiency, R net As a reward for network quality, R move Penalty for moving; β and γ are weighting coefficients that support adjustment and are used to balance the relationship between communication, task and mobility.

7. The method for optimizing embodied intelligent networks in a multidimensional state space according to claim 6, characterized in that: The active optimization framework operates in a periodic decision-making loop to ensure that the embodied agent can respond to environmental changes in real time; the specific steps are as follows: Step S01: Multidimensional state information acquisition and construction of multidimensional state vector S Physical state acquisition: The reinforcement learning scheduler obtains the current physical state information S by calling or subscribing to the robot's operating system ROS perception service. phy This includes location coordinates and movement speed; simultaneously, it acquires related information provided by the task execution module, including the distance to the task target; Network Status Acquisition: The scheduler obtains real-time network status information (S) for all candidate links by calling the robot's ROS communication service interface. net These include signal-to-noise ratio, network latency, and packet loss rate; State vector construction and standardization: This involves constructing and standardizing the acquired physical state information S... phy and real-time network status information S net The data are fused into a multidimensional state vector S; this vector is then Z-score normalized to ensure weight balance across different dimensions of the data, and the normalized vector is then input into the reinforcement learning model. Step S02: Hierarchical Decision Making A through Reinforcement Learning Policy reasoning: Input the standardized multidimensional state vector S into the reinforcement learning co-optimization model; Action Output: The reinforcement learning collaborative optimization model outputs hierarchical actions A = (A...) based on the current optimal policy π(s). comm A phy ); Among them, A comm For communication action information, A phy Information about physical actions; Step S03: Calling and executing ROS services for hierarchical motion robots The reinforcement learning scheduler calls the corresponding Robot Operating System (ROS) service based on the hierarchical action A output, translating the decision into the robot's actual behavior: Communication action execution: If communication action information A comm Including link switching or power adjustment commands, the reinforcement learning scheduler will call the robot's operating system ROS network control service to send configuration update commands to the underlying communication module; Physical action execution: If physical action information A phy For proactive positional movement, the reinforcement learning scheduler will invoke the robot's Robot Operating System (ROS) motion control service, sending the calculated target movement coordinates as the navigation target to the robot's navigation system; the motion control service will then plan the motion trajectory and drive the robot to perform the movement. Step S04: Reward Calculation and Model Optimization Reward Calculation: The reinforcement learning scheduler calculates the reward value of the action in real time based on the new multidimensional state space S' after the action is executed and the preset comprehensive reward function R. Policy optimization: The experience tuple (S, A, R, S') is stored in the experience replay buffer; during training, the reinforcement learning co-optimization model periodically samples data from the buffer and updates the parameters of the deep neural network to achieve continuous policy optimization.

8. A multi-dimensional state-space embodied intelligent network optimization system, characterized in that: To implement the method described in any one of claims 1 to 7, the method seamlessly integrates the construction of a multi-dimensional state space, the construction of a hierarchical action space, and a reinforcement learning collaborative optimization model into the control system of the embodied intelligent agent, and interacts through a specific robot operating system (ROS) interface to transform abstract optimization strategies into concrete physical and communication instructions. Specifically, it includes the following modules: Multidimensional State Awareness Module: Running as a ROS Node in the robot operating system, it is responsible for collecting and processing the physical state information S of the embodied intelligent agent. phy and network status information S net And construct a multidimensional state vector S; Hierarchical Action Module: Running as a Robot Operating System (ROS) Node, it is responsible for outputting control commands to the embodied agent, including communication action information A. comm and physical motion information A phy ; Reinforcement learning scheduler: It runs as an independent robot operating system node (ROS Node), runs the trained reinforcement learning model, performs policy reasoning, and outputs hierarchical actions A. During training, the reinforcement learning co-optimization model periodically samples data from the buffer and updates the parameters of the deep neural network to achieve continuous policy optimization.

9. A multi-dimensional state-space embodied intelligent network optimization computing device, characterized in that: include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by an embodied intelligent network optimization computing device for a multidimensional state space, cause the embodied intelligent network optimization computing device for a multidimensional state space to perform the method according to any one of claims 1 to 7.

Images