A neural cognitive heuristic-based dual-system game visual language navigation method and device

Abstract

The application discloses a dual-system game visual language navigation method and device based on neural cognitive inspiration, which comprises the following steps: acquiring visual feature information and language feature information, extracting visual feature embedding and text feature embedding, and then performing modal alignment and feature fusion to obtain a current state; inputting a state, an action and a global hidden state of a previous moment into a world model based on a gated recurrent unit to generate a current global hidden state; based on the current state and the current global hidden state, based on a multi-brain region collaborative mechanism, generating initial activation tendencies of a goal-oriented system and a habit system by using a prefrontal cortex and a sensory motor cortex, calculating dynamic interest functions of the goal-oriented system and the habit system by using a precingulate gyrus, a thalamus and a premotor cortex, arbitrating and selecting an execution system according to the initial activation tendencies and the dynamic interest functions of the goal-oriented system and the habit system, generating a current execution action by using the execution system, and feeding back and adjusting.

Description

Technical Field

[0001] This application relates to a visual language navigation method and device based on neurocognitive inspiration in a dual-system game, belonging to the field of visual language navigation technology. Background Technology

[0002] Current visual language navigation technologies primarily rely on end-to-end deep reinforcement learning or traditional modular mapping and planning strategies, which have several limitations. First, end-to-end models lack interpretability, implicitly mapping input and output as a "black box," making them prone to long-distance path illusions at complex intersections and susceptible to catastrophic forgetting when fine-tuning in new environments. Second, traditional modular methods are computationally rigid, struggling to bridge the "semantic gap" between natural language commands and visual features, and failing to incorporate the energy-saving mechanism of "practice makes perfect" found in organisms, resulting in high-power computation throughout the day. Third, embodied intelligent product agents driven by large language models suffer from severe "reasoning-action" delays, with generation speeds unable to meet the immediate response requirements of embodied physical interactions. Finally, the decision-making model is singular, lacking a dynamic computational resource allocation and coordination mechanism similar to humans' "relying on intuition in familiar scenes and deliberating deeply in unfamiliar scenes," making it difficult to balance navigation success rate and system energy efficiency. Summary of the Invention

[0003] Objective: In view of at least one of the above technical problems, this application provides a method and apparatus for visual language navigation based on neurocognitive inspiration in dual-system game theory.

[0004] The technical solution adopted in this application is:

[0005] Firstly, this application provides a neurocognitive-inspired dual-system game-theoretic visual-language navigation method, including:

[0006] S1. The agent acquires visual feature information and language feature information containing global natural language path instructions;

[0007] S2. Extract visual feature embeddings and text feature embeddings based on visual feature information and linguistic feature information, respectively. Perform modal alignment and feature fusion on the visual feature embeddings and text feature embeddings to obtain the current state.

[0008] S3. Input the previous state, the previous action, and the previous global hidden state into the world model based on the gated loop unit, and deduce the current global hidden state.

[0009] S4. Based on the current state and the current global latent state, and based on the multi-brain region coordination mechanism, the initial activation tendency of the goal-oriented system and the habit system is generated using the prefrontal cortex and the sensorimotor cortex. The dynamic interest function of the goal-oriented system and the habit system is calculated using the anterior cingulate cortex, thalamus and premotor cortex. The execution system is selected through arbitration based on the initial activation tendency and dynamic interest function of the goal-oriented system and the habit system.

[0010] S5. Based on the current state and the current global hidden state, generate the current execution action using the execution system selected by arbitration;

[0011] S6. After the current action is output, the gamma-aminobutyric acid (GABA) inhibitory neurotransmitter signal transmitted from the basal ganglia to the thalamus is extracted according to the execution system selected by arbitration. This signal is used to regulate the gated loop unit that simulates the thalamic gating mechanism at the next moment. The prediction error of the environmental reward is calculated as a dopamine signal to adjust the parameters of the goal-oriented system.

[0012] Secondly, this application provides a dual-system game-theoretic visual language navigation device based on neurocognitive inspiration, including a processor and a storage medium;

[0013] The storage medium is used to store instructions;

[0014] The processor is configured to operate according to the instructions to execute the method according to the first aspect.

[0015] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0016] Fourthly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0017] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0018] Beneficial Effects: The neurocognitive-inspired dual-system game-theoretic visual-language navigation method and device provided in this application constructs a biomimetic dual-system (habitual system and goal-oriented system) game and arbitration mechanism. Through a closed-loop link of "perception-cognition-decision," it simulates the risk response and regulatory logic of brain circuits. Based on the collaborative arbitration between low-cost intuitive actions triggered by the dynamic balance of neurotransmitters from the sensorimotor cortex to the dorsolateral striatum and long-range planned actions from the prefrontal cortex to the dorsomedial striatum, it achieves efficient integration of real-time obstacle avoidance and global semantic alignment, significantly improving the agent's environmental adaptability, logical transparency, and all-time-domain navigation robustness. It has the following advantages:

[0019] 1. This application constructs a cognitive closed-loop link by simulating the biological regulatory rules of the anterior cingulate cortex and thalamus, and realizes the adaptive switching between high-energy-consuming planning in the prefrontal cortex in unfamiliar and ambiguous scenarios and low-energy-consuming intuitive response in the sensorimotor cortex in familiar scenarios, which significantly improves the efficiency of computing resource allocation and decision-making flexibility of the agent in dynamic environments.

[0020] 2. This application innovatively introduces logical neural network constraints, mapping the abstract high-level decision-making arbitration process into differentiable neural constraint laws, supporting the logical tracing of switching behaviors between habitual systems and goal-oriented systems, and providing a transparent decision-making basis that breaks the "black box" for embodied intelligent navigation systems with high security requirements.

[0021] 3. This application innovatively employs a hybrid density network to construct an independent habitual response path, focusing on its ability to handle multimodal continuous action distributions. Unlike the single-peak action output of traditional reinforcement learning, the network effectively fits the multi-peak uncertainty of the agent's trajectory representation at complex and ambiguous intersections through a Gaussian mixture model, avoiding collisions or deadlocks caused by averaging in traditional strategies.

[0022] 4. This application utilizes a cyclic world model for time-series extrapolation, and combines upper limit confidence interval tree search with Levi-Brown exploration strategy to generate psychological simulation virtual scenarios, breaking through the current limitations of first-person perspective and greatly enhancing the system's ability to avoid and generalize complex long-tail features and dead-end scenarios. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a neurocognitive-inspired dual-system game-theoretic visual language navigation method according to an embodiment of this application. Detailed Implementation

[0024] The present application will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and should not be used to limit the scope of protection of the present application.

[0025] In the description of this application, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0026] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0027] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0028] Example 1: This example provides a dual-system game-theoretic visual-language navigation method inspired by neurocognition, such as... Figure 1 As shown, it includes:

[0029] S1. The agent acquires visual feature information as well as language feature information containing global natural language path instructions.

[0030] The visual feature information is obtained by acquiring RGB images from a frontal first-view perspective using the onboard visual sensor. The visual feature information includes the color, texture, and spatial layout of the current local scene.

[0031] In this embodiment, the global natural language path instructions in the language feature information are driven by a continuous environment expert instruction set.

[0032] S2. Extract visual feature embeddings and text feature embeddings based on visual feature information and linguistic feature information respectively. Perform modal alignment and feature fusion on the visual feature embeddings and text feature embeddings to obtain the current state.

[0033] In this embodiment, a pre-trained image-text model is used to extract visual feature embeddings and text feature embeddings. For example, the pre-trained image-text model uses a pre-trained CLIP encoder.

[0034] In some embodiments, modal alignment and feature fusion are performed on visual feature embeddings and text feature embeddings to obtain the current state, including:

[0035] Cross-modal aligned features are obtained by embedding textual features into query visual features using a multi-head cross-attention mechanism and performing cross-modal deep alignment, as shown below:

[0036] ;

[0037] , , ;

[0038] in, For cross-modal alignment features, Let Q be the normalized exponential function, and let Q, K, and V be the query matrix, key matrix, and value matrix of the multi-head cross-attention mechanism, respectively. , , These are the weights of the query matrix, key matrix, and value matrix, respectively. Scaling factor , These are visual feature embedding and text feature embedding, respectively.

[0039] After cross-modal alignment features and text feature embeddings are processed through residual connections, feedforward neural networks, and layer normalization to compensate for local biases, mean pooling is performed along the sequence dimension to obtain the current state, represented as:

[0040] ;

[0041] ;

[0042] ;

[0043] in, As the first intermediate feature, For layer normalization operation, As the second intermediate feature, It is a feedforward neural network. For mean pooling operation, Let t represent the current state and t represent the current time.

[0044] S3. Input the previous state, the previous action, and the previous global hidden state into the world model based on the gated loop unit, and deduce the current global hidden state.

[0045] This step is represented as:

[0046] ;

[0047] in, This is the current global hidden state. For the gated loop unit of the world model, A multilayer perceptron network for a world model. Indicates feature splicing, This refers to the state at the previous moment. The action in the previous moment. This represents the global hidden state at the previous time step, where t and t-1 represent the current time step and the previous time step, respectively.

[0048] S4. Based on the current state and the current global latent state, and based on the multi-brain region coordination mechanism, the initial activation tendency of the goal-oriented system and the habit system is generated using the prefrontal cortex and the sensorimotor cortex. The dynamic interest function of the goal-oriented system and the habit system is calculated using the anterior cingulate cortex, thalamus and premotor cortex. The execution system is selected through arbitration based on the initial activation tendency and dynamic interest function of the goal-oriented system and the habit system.

[0049] In some embodiments, the initial activation tendency of the goal-oriented system and habit system generated by utilizing the prefrontal cortex and sensorimotor cortex is represented as follows:

[0050] ;

[0051] ;

[0052] in, , These are the initial activation tendencies of the goal-oriented system and the habitual system, respectively. For the prefrontal cortex based on the current state With the current global hidden state The external task intent being analyzed. For the prefrontal cortex based on the current state With the current global hidden state Analysis of the exploration intention based on the intrinsic motivation of maximum entropy. For the sensorimotor cortex based on the current state With the current global hidden state The extracted scene familiarity features; t represents the current time, PFC represents the prefrontal cortex, and SMC represents the sensorimotor cortex.

[0053] In this embodiment, the network loss functions for the prefrontal cortex (PFC) and sensorimotor cortex (SMC) are as follows:

[0054] ;

[0055] ;

[0056] in, , These are the optimized loss functions for the prefrontal cortex and the sensorimotor cortex, respectively. , , These are the weighting coefficients, The external task intent value is given by the underlying value network. The value of the exploration intention is based on the maximum entropy intrinsic motivation given by the underlying value network. The maximum cosine similarity score is calculated between the current state and historical states in the historical successful trajectory feature library. For the correlation penalty loss of the logistic neural network, positive=True indicates that a positive correlation between physical and logical components is required. It is the square of the L2 norm.

[0057] In some embodiments, the dynamic interest functions of the goal orientation system and habit system are calculated using the anterior cingulate cortex, thalamus, and premotor cortex, and are expressed as follows:

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] ;

[0063] in, , These are the dynamic interest functions for the goal-oriented system and the habitual system, respectively. , The anterior cingulate cortex is based on the current state. With the current global hidden state The output gain for goal-oriented systems and habitual systems. It is a non-linear activation function. To simulate a multi-layer sensor network for crisis monitoring of the anterior fascia, Indicates feature splicing, The gain is due to the release of the dorsomedial thalamic nucleus into the anterior frontal lobe. This refers to the increase in the release of energy from the ventroanterior / ventrolateral nuclei of the thalamus to the sensorimotor cortex. A gated recurrent unit that simulates the gating mechanism of the thalamus. The data transmitted from the basal ganglia to the thalamus at the previous time t-1. α-aminobutyric acid (GABA) is an inhibitory neurotransmitter signal. This refers to the internal hidden state memory of the thalamic gating network in the previous time step. This represents the active inhibitory signal released by the premotor cortex against the habitual system. To simulate the inhibitory function of the anterior motor cortex, a multilayer perceptron network is used, where ACC represents the anterior cingulate cortex. It represents the thalamus, and PMC represents the premotor cortex.

[0064] In this embodiment, the anterior cingulate cortex (ACC) and thalamus ( The network loss function is as follows:

[0065] ;

[0066] ;

[0067] ; ;

[0068] ;

[0069] ;

[0070] in, , These are the logical constraint loss functions for generating the target-oriented gain and the habitual gain in the forward camber, respectively. For the temporal and physical polarity constraint loss function of the thalamic gating network, Logical-physical constraint loss function for the premotor cortex to perform the inhibition network branch; To control the loss weight coefficient of the logical constraint strength, This is a comprehensive crisis signal summarized by the ACC real-time monitoring of the anterior fascia. The correlation penalty loss is used for logistic neural networks. `positive=True` indicates a positive physical-logic correlation is required, and `positive=False` indicates a negative physical-logic correlation is required. The square of the L2 norm. , This represents the internal hidden state memory of the thalamic gating network at the current time t and the previous time. , These represent the internal hidden memories of the dorsomedial thalamus at the current time t and the previous time, respectively. , These are the internal hidden memories of the anterior / ventrolateral thalamic nuclei at the current time t and the previous time, respectively. For the j-th nucleus in the thalamic network to receive the basal ganglia Gated layer connection weights for GABA (aminobutyric acid) inhibitory neurotransmitter signaling. The penalty weight coefficient for time series smoothing constraints, This represents the mathematical expectation over the entire trajectory time step. VAVL and VAVL represent the dorsomedial thalamic nucleus and the ventroanterior / ventrolateral thalamic nucleus, respectively, and j represents the j-th nucleus.

[0071] In some embodiments, arbitration is used to select an execution system based on the initial activation tendency of the goal-oriented system, the habitual system, and a dynamic interest function, including:

[0072] ;

[0073] ;

[0074] ;

[0075] in, , They represent the habit system and the goal-oriented system, respectively. , These are the selection probability values ​​of the target-oriented system and the habitual system output by the meta-controller, respectively. It is an exponential function. , These are the initial activation tendencies of the goal-oriented system and the habitual system, respectively. , These are the dynamic interest functions for the goal-oriented system and the habitual system, respectively. The enforcement system selected for arbitration is chosen from either a customary system or a goal-oriented system; Sampling is performed for the category distribution.

[0076] In this embodiment, the loss function of the meta controller is as follows:

[0077] ;

[0078] ;

[0079] in, Let be the loss function of the meta-controller. The enforcement system selected for arbitration is chosen from the customary system. or goal-oriented system , Selected from or , , These are the selection probability values ​​of the target-oriented system and the habitual system output by the meta-controller, respectively. Represents the mathematical expectation. The meta controller is based on the current state With the current global hidden state Select the dominant function of the execution system. The penalty coefficient for switching costs, To switch cost parameters, These are the weighting coefficients for logical constraints. The total loss of the global logistic neural network is defined as the loss function of the logistic neural network in the prefrontal cortex, sensorimotor cortex, anterior cingulate cortex, thalamus, premotor cortex, and basal ganglia output nuclei.

[0080] S5. Based on the current state and the current global hidden state, generate the current execution action using the execution system selected by arbitration.

[0081] S51. If the arbitration selects the customary system, the customary system, based on the current state and the current global hidden state, fits a Gaussian mixture distribution of the continuous action space through a mixture density network, and generates the current action using a maximum weight mean extraction strategy, as follows:

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] ;

[0087] in, For the habit system in the current state and the current global hidden state Output the currently executing action. The probability distribution; t represents the current time. , , Here, represents the selection probability weight, distribution mean, and standard deviation of the k-th Gaussian component output by the mixed density network, respectively, where K is the total number of Gaussian components. It is a normal distribution function. Select the index of the Gaussian component with the highest probability weight from the K Gaussian components. To select the Gaussian component with the highest probability weight The distribution mean, Let be the continuous yaw angle, and tanh be the hyperbolic tangent activation function. For continuous movement ratio, It is a non-linear activation function. For a multilayer perceptron network of a habit system, Indicates feature splicing;

[0088] S52. If the arbitration selects a goal-oriented system, the goal-oriented system, based on the current state and the current global hidden state, uses the upper confidence interval tree search algorithm to conduct multi-step psychological simulations in the latent space of the world model, and combines the Levi-Brown exploration mechanism to generate the current action to be executed, including:

[0089] Before executing the upper confidence interval tree search algorithm to conduct multi-step psychological simulations in the latent space of the world model, the computational resource scale and exploration tendency of the goal-oriented system are determined by the basic parameters. Initialization, and inference depth influenced by feedback from the previous time step. , breadth of deduction and dynamic weights Common constraints and controls;

[0090] ;

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] in, , , Here, represents the selection probability weight, distribution mean, and standard deviation of the k-th Gaussian component output by the mixed density network, respectively, where K is the total number of Gaussian components. It is a normal distribution function. Select the index of the Gaussian component with the highest probability weight from the K Gaussian components. Select the index of the Gaussian component with the highest probability weight from the K Gaussian components. To select the Gaussian component with the highest probability weight The distribution mean, To provide the continuous yaw angle output by the system, tanh is the hyperbolic tangent activation function. To accommodate the continuous movement ratio output by the system, To adapt to the execution actions output by the system, It is a non-linear activation function. For a multilayer perceptron network of a habit system, Indicates feature splicing, In order to be in Injecting Levi flight long tail noise on the basis With Brownian noise The generated exploratory planning actions, The dynamic weights fed back at the previous time step t-1; For the habit system in the current state and the current global hidden state The following are exploratory planning actions The given prior probability evaluation value; For the currently executing action, To plan actions based on exploration The global hidden state at the next moment is derived using the world model. virtual state for the next moment and the global hidden state at the next moment The overall assessment value These are the weighting coefficients. The total number of visits to the current node during the mental simulation process. To perform exploratory planning actions at the current node The number of visits to the corresponding virtual child nodes. The virtual state of the next moment in the world model derived by the underlying value network. and the global hidden state at the next moment The initial evaluation value, The reward weighting coefficient for semantic progress difference. virtual state for the next moment and current state Semantic progress difference in task instructions, To prevent the penalty weighting coefficient from looping, For indicator functions, It represents the set of hidden states that an intelligent agent has traversed in the real physical world. Let i be the i-th historical hidden state. For cosine similarity, As a safety threshold, For the gated loop unit of the world model, A multilayer perceptron network for a world model. For prior prediction of multilayer perceptron networks.

[0101] S6. After the current action is output, the data transmitted from the basal ganglia to the thalamus is extracted according to the execution system selected by the arbitration. The α-aminobutyric acid (GABA) inhibitory neurotransmitter signal feeds back to regulate the gated loop unit of the thalamic gating mechanism in the next time step; and calculates the prediction error of environmental reward as a dopamine signal to regulate the parameters of the goal-oriented system.

[0102] In some embodiments, the execution system selected by the arbitration extracts data transmitted from the basal ganglia back to the thalamus. - GABA inhibitory neurotransmitter signaling includes:

[0103] ;

[0104] ;

[0105] ;

[0106] in, The data transmitted from the basal ganglia to the thalamus at the current time t. α-aminobutyric acid (GABA) is an inhibitory neurotransmitter signal. Indicates feature splicing, This represents the concentration of inhibitory neurotransmitters projected from the basal ganglia output nucleus to the dorsomedial thalamic nucleus at the current time t. This represents the concentration of inhibitory neurotransmitters projected from the basal ganglia output nucleus to the ventroanterior / ventrolateral thalamic nucleus at the current time t. , These are the logistic neural networks that project from the basal ganglia output nuclei to the dorsomedial thalamic nuclei and the ventroanterior / ventrolateral thalamic nuclei. It is a smooth nonlinear activation function; The enforcement system selected for arbitration is chosen from either a customary system or a goal-oriented system.

[0107] In some embodiments, the prediction error of the calculated environmental reward is used as a dopamine signal to adjust the parameters of the goal-oriented system, including:

[0108] ;

[0109] ;

[0110] in, A crisis intensity index derived from the normalized mapping of prediction errors for environmental rewards. This is a truncation function. Prediction error of environmental reward within the time window The average value, , These represent the maximum and minimum values ​​of the prediction error for environmental rewards, respectively. For the adaptive adjustment function of the goal-oriented system, The dynamic weights fed back at the current time t. , The depth and breadth of the deduction are fed back at the current time t. These are the basic parameters.

[0111] , , This is used to update the dynamic weights, inference depth, and inference breadth of the target-oriented system for the next time step, thereby achieving adaptive allocation of computing power.

[0112] At the level of physical constraints on the brain's neurons, to ensure that the basal ganglia execute the strict biological logic of "lateral inhibition," that is, when the meta-controller activates a certain execution system, it must release a low concentration of [something] downstream of that execution system. -Aminobutyric acid (GABA) to relieve inhibition and release high concentrations in the competing system. γ-aminobutyric acid (GABA) is used to strongly inhibit the logistic anchoring loss of the basal ganglia output nuclei. In this embodiment, the loss function for the basal ganglia output nuclei (globus pallidus medialis GPi / substantia nigra reticularis SNr) is as follows:

[0113] ;

[0114] in, The logical constraint loss function for the output nuclei of the basal ganglia.

[0115] Example 2: Based on Example 1, this example provides a dual-system game-theoretic visual language navigation device inspired by neurocognition, including a processor and a storage medium;

[0116] The storage medium is used to store instructions;

[0117] The processor is configured to operate according to the instructions to execute the method according to Embodiment 1.

[0118] Example 3: Based on Example 1, this example provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Example 1.

[0119] Example 4: Based on Example 1, this example provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in Example 1.

[0120] Example 5: Based on Example 1, this example provides a computer program product, including a computer program that, when executed by a processor, implements the method described in Example 1.

[0121] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A dual-system game-theoretic visual-language navigation method inspired by neurocognition, characterized in that, include: S1. The agent acquires visual feature information and language feature information containing global natural language path instructions; S2. Extract visual feature embeddings and text feature embeddings based on visual feature information and linguistic feature information, respectively. Perform modal alignment and feature fusion on the visual feature embeddings and text feature embeddings to obtain the current state. S3. Input the previous state, the previous action, and the previous global hidden state into the world model based on the gated loop unit, and deduce the current global hidden state. S4. Based on the current state and the current global latent state, and based on the multi-brain region coordination mechanism, the initial activation tendency of the goal-oriented system and the habit system is generated using the prefrontal cortex and the sensorimotor cortex. The dynamic interest function of the goal-oriented system and the habit system is calculated using the anterior cingulate cortex, thalamus and premotor cortex. The execution system is selected through arbitration based on the initial activation tendency and dynamic interest function of the goal-oriented system and the habit system. S5. Based on the current state and the current global hidden state, generate the current execution action using the execution system selected by arbitration; S6. After the current action is output, extract the data transmitted from the basal ganglia to the thalamus according to the execution system selected by the arbitration. The α-aminobutyric acid (GABA) inhibitory neurotransmitter signal feeds back to regulate the gated loop unit of the thalamic gating mechanism in the next time step; and calculates the prediction error of environmental reward as a dopamine signal to regulate the parameters of the goal-oriented system.

2. The method according to claim 1, characterized in that, The current state is obtained by modal alignment and feature fusion of visual feature embeddings and text feature embeddings, including: Cross-modal aligned features are obtained by embedding textual features into query visual features using a multi-head cross-attention mechanism and performing cross-modal deep alignment, as shown below: ； ， ， ； in, For cross-modal alignment features, Let Q be the normalized exponential function, and let Q, K, and V be the query matrix, key matrix, and value matrix of the multi-head cross-attention mechanism, respectively. , , These are the weights of the query matrix, key matrix, and value matrix, respectively. Scaling factor , These are visual feature embedding and text feature embedding, respectively. After cross-modal alignment features and text feature embeddings are processed through residual connections, feedforward neural networks, and layer normalization to compensate for local biases, mean pooling is performed along the sequence dimension to obtain the current state, represented as: ； ； ； in, As the first intermediate feature, For layer normalization operation, As the second intermediate feature, It is a feedforward neural network. For mean pooling operation, Let t represent the current state and t represent the current time.

3. The method according to claim 1, characterized in that, Step S3 is represented as follows: ； in, This is the current global hidden state. For the gated loop unit of the world model, A multilayer perceptron network for a world model. Indicates feature splicing, This refers to the state at the previous moment. The action in the previous moment. This represents the global hidden state at the previous time step, where t and t-1 represent the current time step and the previous time step, respectively.

4. The method according to claim 1, characterized in that, The initial activation tendency of the goal-oriented system and habit system generated by utilizing the prefrontal cortex and sensorimotor cortex is represented as: ； ； in, , These are the initial activation tendencies of the goal-oriented system and the habitual system, respectively. For the prefrontal cortex based on the current state With the current global hidden state The external task intent being analyzed. For the prefrontal cortex based on the current state With the current global hidden state Analysis of the exploration intent based on the intrinsic motivation of maximum entropy. For the sensorimotor cortex based on the current state With the current global hidden state Extracted scene familiarity features; t represents the current time, PFC represents the prefrontal cortex, and SMC represents the sensorimotor cortex; And / or, using the anterior cingulate cortex, thalamus, and premotor cortex, calculate the dynamic interest functions of the goal-oriented system and habitual system, expressed as: ； ； ； ； ； in, , These are the dynamic interest functions for the goal-oriented system and the habitual system, respectively. , The anterior cingulate cortex is based on the current state. With the current global hidden state The output gain for goal-oriented systems and habitual systems. It is a non-linear activation function. To simulate a multi-layer sensor network for crisis monitoring of the anterior fascia, Indicates feature splicing, The gain is due to the release of the dorsomedial thalamic nucleus into the anterior frontal lobe. This refers to the increase in the release of energy from the ventroanterior / ventrolateral nuclei of the thalamus to the sensorimotor cortex. A gated recurrent unit that simulates the gating mechanism of the thalamus. The data transmitted from the basal ganglia to the thalamus at the previous time t-1. α-aminobutyric acid (GABA) is an inhibitory neurotransmitter signal. This refers to the internal hidden state memory of the thalamic gating network in the previous time step. This represents the active inhibitory signal released by the premotor cortex against the habitual system. To simulate the inhibitory function of the anterior motor cortex, a multilayer perceptron network is used, where ACC represents the anterior cingulate cortex. The thalamus represents the brain, and PMC represents the premotor cortex. And / or, the execution system is selected through arbitration based on the initial activation tendency of the goal-oriented system, the habitual system, and the dynamic interest function, including: ； ； ； in, , They represent the habit system and the goal-oriented system, respectively. , These are the selection probability values ​​of the target-oriented system and the habitual system output by the meta-controller, respectively. It is an exponential function. , These are the initial activation tendencies of the goal-oriented system and the habitual system, respectively. , These are the dynamic interest functions for the goal-oriented system and the habitual system, respectively. The enforcement system selected for arbitration is chosen from either a customary system or a goal-oriented system; Sampling is performed for the category distribution.

5. The method according to claim 1, characterized in that, Step S5 includes: S51. If the arbitration selects the customary system, the customary system, based on the current state and the current global hidden state, fits a Gaussian mixture distribution of the continuous action space through a mixture density network, and generates the current action using a maximum weight mean extraction strategy, as follows: ； ； ； ； ； in, For the habit system in the current state and the current global hidden state Output the currently executing action. The probability distribution; t represents the current time. , , Here, represents the selection probability weight, distribution mean, and standard deviation of the k-th Gaussian component output by the mixed density network, respectively, where K is the total number of Gaussian components. It is a normal distribution function. Select the index of the Gaussian component with the highest probability weight from the K Gaussian components. To select the Gaussian component with the highest probability weight The distribution mean, Let be the continuous yaw angle, and tanh be the hyperbolic tangent activation function. For continuous movement ratio, It is a non-linear activation function. For a multilayer perceptron network of a habit system, Indicates feature splicing; S52. If the arbitration selects a goal-oriented system, the goal-oriented system, based on the current state and the current global hidden state, uses the upper confidence interval tree search algorithm to conduct multi-step psychological simulations in the latent space of the world model, and combines the Levi-Brown exploration mechanism to generate the current action to be executed, including: Before executing the upper confidence interval tree search algorithm to conduct multi-step psychological simulations in the latent space of the world model, the computational resource scale and exploration tendency of the goal-oriented system are determined by the basic parameters. Initialization, and inference depth influenced by feedback from the previous time step. Breadth of deduction and dynamic weights Common constraints and controls; ； ； ； ； ； ； ； ； ； ； in, , , Here, represents the selection probability weight, distribution mean, and standard deviation of the k-th Gaussian component output by the mixed density network, respectively, where K is the total number of Gaussian components. It is a normal distribution function. Select the index of the Gaussian component with the highest probability weight from the K Gaussian components. Select the index of the Gaussian component with the highest probability weight from the K Gaussian components. To select the Gaussian component with the highest probability weight The distribution mean, To provide the continuous yaw angle output by the system, tanh is the hyperbolic tangent activation function. To accommodate the continuous movement ratio output by the system, To adapt to the execution actions output by the system, It is a non-linear activation function. For a multilayer perceptron network of a habit system, Indicates feature splicing, In order to be in Injecting Levi flight long tail noise on the basis With Brownian noise The generated exploratory planning actions, The dynamic weights fed back at the previous time step t-1; For the habit system in the current state and the current global hidden state The following are exploratory planning actions The given prior probability evaluation value; For the currently executing action, To plan actions based on exploration The global hidden state at the next moment is derived using the world model. virtual state for the next moment and the global hidden state at the next moment The overall assessment value These are the weighting coefficients. The total number of visits to the current node during the mental simulation process. To perform exploratory planning actions at the current node The number of visits to the corresponding virtual child nodes. The virtual state of the next moment in the world model derived by the underlying value network. and the global hidden state at the next moment The initial evaluation value, The reward weighting coefficient for semantic progress difference. virtual state for the next moment and current state Semantic progress difference in task instructions To prevent the penalty weighting coefficient from looping, For indicator functions, It represents the set of hidden states that an intelligent agent has traversed in the real physical world. Let i be the i-th historical hidden state. For cosine similarity, As a safety threshold, For the gated loop unit of the world model, A multilayer perceptron network for a world model. For prior prediction of multilayer perceptron networks.

6. The method according to claim 1, characterized in that, The execution system selected by the arbitration extracts data transmitted from the basal ganglia back to the thalamus. - GABA inhibitory neurotransmitter signaling includes: ； ； ； in, The data transmitted from the basal ganglia to the thalamus at the current time t. α-aminobutyric acid (GABA) is an inhibitory neurotransmitter signal. Indicates feature splicing, This represents the concentration of inhibitory neurotransmitters projected from the basal ganglia output nucleus to the dorsomedial thalamic nucleus at the current time t. The concentration of inhibitory neurotransmitters projected from the basal ganglia output nucleus to the ventroanterior / ventrolateral thalamic nucleus at the current time t. , These are the logistic neural networks that project from the basal ganglia output nuclei to the dorsomedial thalamic nucleus and the ventroanterior / ventrolateral thalamic nuclei. It is a smooth nonlinear activation function; The enforcement system selected for arbitration is chosen from either a customary system or a goal-oriented system.

7. The method according to claim 1, characterized in that, The prediction error of the calculated environmental reward is used as a dopamine signal to adjust the parameters of the goal-oriented system, including: ； ； in, A crisis intensity index derived from the normalized mapping of prediction errors for environmental rewards. This is a truncation function. Prediction error of environmental reward within the time window The average value, , These represent the maximum and minimum values ​​of the prediction error for environmental rewards, respectively. For the adaptive adjustment function of the goal-oriented system, The dynamic weights fed back at the current time t. , The depth and breadth of the deduction are fed back at the current time t. These are the basic parameters.

8. A dual-system game-theoretic visual-language navigation device inspired by neurocognition, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

Images