Hierarchical strategy method based on awareness guidance under sensing sparsity

Abstract

The invention discloses a hierarchical strategy method based on Awareness guidance under sensing sparsity, and the method comprises the steps: completing the sampling of sub-target nodes through a sub-target node construction method based on Awareness data, and improving the overall learning efficiency; reinforcement learning is modeled into an undirected probability graph model, the concept of edge cutting and point cutting is introduced, an index system of sub-target sampling is included, a Zhixiao coordinate set obtained through sampling is selected through a path optimization method, and the model can effectively recognize and utilize key information in an environment structure; according to the method, strategies on different levels are ensured not to generate conflicts, so that the learning efficiency and the strategy quality are remarkably improved in a sparse reward environment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a hierarchical strategy method based on knowledge guidance under sparse perception. Background Art

[0002] Currently, intelligent agents face the problem of sparse perception, meaning that the reward signal they receive in most states is close to zero. This results in low learning efficiency for traditional reinforcement learning algorithms. Hierarchical strategies based on knowledge and trust guidance face technical challenges such as incomplete and uncertain information, establishing and maintaining trust metrics, representing and managing knowledge, detecting novelty, and improving computational efficiency and real-time processing capabilities. Furthermore, attention must be paid to data security and privacy protection. Summary of the Invention

[0003] In order to solve the above problems in the prior art, the present invention proposes a hierarchical strategy method based on knowledge guidance under perception sparsity, which solves the problem in the prior art that the existing multi-agent learning algorithm has low learning efficiency under perception sparsity.

[0004] The present invention provides a layered strategy method based on information guidance under sparse perception, comprising the following steps: (1) Setting the state-target mapping function ; Upper-layer strategy update frequency , total number of training iterations N, frequency of lower-level strategy updates , training batch size B, number of coordinates ; Initialize upper and lower layer strategy parameters , and create playback buffer B; (2) Environmental modeling and information coordinate selection, including: (21) Modeling the reinforcement learning environment as a probabilistic undirected graph model, where nodes represent states and edges represent transition probabilities between states; (22) Collect key information during the environmental exploration process, including key nodes in the environment; (23) generating a set of information coordinates based on the information data, wherein the information coordinates serve as key information points for sampling sub-targets; (3) Iterative training and updating, including: (31) Through upper-level strategies Generate subgoals , and assign sub-goals to lower-level strategies; (32) Lower-level strategies According to sub-goals Execute an action , get the new status and rewards ; (33) The quadruple Store in playback buffer B and calculate The novelty will Join the novelty-based priority queue U; (4) Strategy update: sampling from replay buffer B samples, sampled from the priority queue U Highly novel samples, combined with reinforcement learning algorithm updates .

[0005] Furthermore, step (1) includes the following steps: (11) Reset the environment and sample the initial state , initialize the upper and lower layer strategy parameters ; (12) Create a playback buffer B to store the quad ; (13) Create a priority queue U whose elements are sorted by the novelty of the state, which is used to optimize the policy update.

[0006] Furthermore, in step (21), the construction of the probabilistic undirected graph model includes: based on the environmental exploration data, identifying the transition probability between states, and extracting key nodes through the edge-cutting algorithm, and using the key nodes as a candidate set for sub-target sampling.

[0007] Furthermore, in step (31), sub-goals are generated The details are as follows: According to the current status and the coordinate set of the information, through the upper-level strategy Sampling .

[0008] Furthermore, in step (32), the lower-level strategy performs the action The process includes: according to the current state and sub-goals , through strategy Select Action , and execute to update the environment state.

[0009] Furthermore, in step (4), the specific operation of the strategy update is: randomly extract from the playback buffer B samples, select the top samples from the priority queue U by novelty order states, and jointly optimized by gradient descent .

[0010] The novelty calculation process is as follows: First, data is collected and preprocessed to build or update a knowledge base containing historical information. Next, a trust metric is defined based on the reliability and consistency of the information source. Similarity calculations, distance metrics, and anomaly detection techniques are then used to evaluate the differences between the new information and existing knowledge. A novelty score is then assigned based on the trustworthiness. A threshold is set to filter sufficiently novel information for further analysis or inclusion in the knowledge base, and the entire process is optimized through a feedback loop.

[0011] Furthermore, the specific process based on a graph model is as follows: First, entities and their relationships are represented as nodes and edges to construct a graph model. The paths and connections in the graph are used to define knowledge and trust propagation rules, and the trust metrics of the nodes are updated through a message passing algorithm. For novelty calculation, the similarity of new information with existing nodes in the graph is compared and its uniqueness is evaluated. A threshold is set to identify nodes with high novelty, which can then be incorporated into the existing graph structure or treated as new knowledge points.

[0012] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the computer program is loaded into the processor, it implements any one of the hierarchical strategy methods based on knowledge guidance under perceptual sparsity.

[0013] The present invention provides a storage medium storing a computer program, which, when executed by a processor, implements any one of the aforementioned methods for layered strategies based on knowledge guidance under perceptual sparsity.

[0014] Beneficial Effects: Compared with existing technologies, this invention has the following advantages: By breaking down complex tasks into multiple levels of subtasks and utilizing information data as key nodes, it guides intelligent agents to conduct effective exploration and utilization. Furthermore, through multi-source and sparse sensing technologies, it optimizes the use of sensing resources and improves the decision-making ability of intelligent agents in information-deficient situations. Furthermore, the invention utilizes distributed computing and wireless communication technologies to support the collaborative work of multiple intelligent agents, ensuring reliable information transmission and efficient decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are only used to illustrate and explain the present invention, and do not limit the scope of implementation of the present invention.

[0017] like Figure 1As shown, an embodiment of the present invention provides a layered strategy method based on information guidance under sparse perception, including the following steps: Step 1: Set the state-target mapping function ; Upper-layer strategy update frequency , total number of training iterations N, frequency of lower-level strategy updates , training batch size B, number of coordinates ; Initialize upper and lower layer strategy parameters , and create a playback buffer B; use an iterative method to train. It includes the following steps: Step 1-1. Reset the environment and sample the initial state , initialize the upper and lower layer strategy parameters and playback buffer B; where Used for upper-level strategies, responsible for selecting sub-goals; Used for lower-level strategies, responsible for executing subtasks.

[0018] Step 1-2: Create playback buffer B to store quadruple ; Steps 1-3: Create a priority queue U whose elements are sorted by the novelty of the state, which is used to optimize the policy update; the elements in the first policy are sorted according to their novelty (i.e., exploratory).

[0019] Step 2: Environment modeling and knowledge coordinate selection, including environment modeling, knowledge data collection, and knowledge coordinate generation. Environment modeling models the reinforcement learning environment into a probabilistic undirected graph model, where each node represents a state and edges represent transition probabilities between states. Knowledge data collection involves collecting key knowledge data during the environment exploration process, including key nodes in the environment (such as important landmarks, obstacles, etc.). Knowledge coordinate generation involves generating a set of knowledge coordinates based on the collected knowledge data. As key information points in the environment, knowledge coordinates can provide important knowledge guidance. This includes the following steps: Step 2-1: Use upper-level strategies Sampling the known coordinates and generating pseudo coordinates ; Step 2-2, generate sub-goals .

[0020] Step 3: Iterative training and updating of playback buffer and priority queue. The upper-level strategy generates sub-goals and the lower-level strategy executes actions to achieve iterative training. The upper-level strategy is mainly responsible for selecting and assigning sub-goals. The lower-level strategy selects actions and executes them according to the assigned sub-goals. Store in replay buffer B and calculate the new state The novelty of Add to the priority queue U. Includes the following steps: Step 3-1: Upper-level strategy Formulate sub-goals, select and allocate sub-goals, and the upper-level strategy is based on the current state And the knowledge coordinate set, select a sub-target , the selected sub-goal Assigned to lower-level policies.

[0021] Step 3-2: Lower-level strategy Achieve sub-goals, select actions and execute actions; the lower-level strategy is based on the current state and sub-goals , select an action , perform the selected action , and observe the new state returned by the environment and rewards .

[0022] Step 3-3. Select and execute an action , observe the new state and rewards , update B and the novelty-based priority queue U.

[0023] Step 4: Strategy update: randomly sample a batch of samples from the replay buffer B, use the reinforcement learning algorithm to update the upper and lower strategies, select the state with the highest novelty in the priority queue U, and conduct additional exploration and learning to optimize the strategy update process. Coordinates, combined with priority queue U update .

Claims

1. A hierarchical strategy method based on knowledge guidance under sparse perception, characterized by: The following steps are involved: (1) Setting the state-target mapping function ; Upper-layer strategy update frequency , total number of training iterations N, frequency of lower-level strategy updates , training batch size B, number of coordinates ; Initialize upper and lower layer strategy parameters , and create playback buffer B; (2) Environmental modeling and information coordinate selection, including: (21) Modeling the reinforcement learning environment as a probabilistic undirected graph model, where nodes represent states and edges represent transition probabilities between states; (22) Collect key information during the environmental exploration process, including key nodes in the environment; (23) generating a set of information coordinates based on the information data, wherein the information coordinates serve as key information points for sampling sub-targets; (3) Iterative training and updating, including: (31) Through upper-level strategies Generate subgoals , and assign sub-goals to lower-level strategies; (32) Lower-level strategies According to sub-goals Execute an action , get the new status and rewards ; (33) The quadruple Store in playback buffer B and calculate The novelty will Join the novelty-based priority queue U; (4) Strategy update: sampling from replay buffer B samples, sampled from the priority queue U Highly novel samples, combined with reinforcement learning algorithm updates .

2. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: Step (1) includes the following steps: (11) Reset the environment and sample the initial state , initialize the upper and lower layer strategy parameters ; (12) Create a playback buffer B to store the quad ; (13) Create a priority queue U whose elements are sorted by the novelty of the state, which is used to optimize the policy update.

3. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: In step (21), the construction of the probabilistic undirected graph model includes: based on the environmental exploration data, identifying the transition probability between states, and extracting key nodes through the edge-cutting algorithm, and using the key nodes as the candidate set for sub-target sampling.

4. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: In step (31), generate sub-goals The details are as follows: According to the current status and the coordinate set of the information, through the upper-level strategy Sampling .

5. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: In step (32), the lower-level strategy performs the action The process includes: according to the current state and sub-goals , through strategy Select Action , and execute to update the environment state.

6. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: In step (4), the specific operation of the strategy update is: randomly extract from the playback buffer B samples, select the top samples from the priority queue U by novelty order states, and jointly optimized by gradient descent .

7. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: The novelty calculation process is as follows: First, collect and preprocess data to build or update a knowledge base containing historical information; then, define a trust metric based on the reliability and consistency of the information source; then, use similarity calculation, distance measurement and anomaly detection to evaluate the difference between new information and existing knowledge, and give a novelty score based on the trust level; set a threshold to filter out sufficiently novel information for further analysis or inclusion in the knowledge base, and optimize the entire process through a feedback loop.

8. The layered strategy method based on knowledge guidance under perceptual sparsity according to claim 1 is characterized in that: The specific process based on the graph model is as follows: First, entities and their relationships are represented as nodes and edges to construct a graph model; the paths and connections in the graph are used to define knowledge and trust propagation rules, and the trust metrics of the nodes are updated through a message passing algorithm; For novelty calculation, the similarity of new information with existing nodes in the graph is compared and its uniqueness is evaluated; A threshold is set to identify nodes with high novelty, which are then incorporated into the existing graph structure or used as new knowledge points.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into a processor, it implements a layered strategy method based on knowledge guidance under perceptual sparsity according to any one of claims 1 to 8.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements a layered strategy method based on knowledge guidance under perceptual sparsity according to any one of claims 1 to 8.

Images