Visual reasoning method based on reflection mechanism working memory

Abstract

The invention provides a visual reasoning method based on reflection mechanism working memory. The visual reasoning method comprises a visual feature analysis step, a working memory coding step, a reasoning step and a result output step. According to the invention, work memory optimization based on a reflection mechanism is realized through work memory noise addition and long-term memory guide denoising, and effective interaction between work memory and long-term memory is realized, so that the work memory can be self-adjusted and corrected by using stable knowledge in the long-term memory; the long-term stability and the information retention capability of the system are enhanced, so that the system maintains stable memory representation in the visual reasoning task and is not easily interfered, and the self-adjusting capability in the visual reasoning task process is enhanced. A flow matching network is introduced to realize a denoising process, and an optimization process of a working memory is constructed into a continuous-time dynamic system, so that the working memory can be gradually recovered to a noiseless state along an optimal track, and the robustness and adaptability of memory representation are enhanced.

Description

Technical Field

[0001] The image processing technology of this invention specifically relates to visual reasoning technology based on reflective working memory. Background Technology

[0002] Existing artificial intelligence systems often fail to perform well in working memory under complex and dynamic environments when dealing with complex visual reasoning tasks. The main problems are as follows: First, working memory is easily affected by environmental interference and becomes unstable; second, it lacks self-regulation and optimization mechanisms and cannot perform self-reflection and memory reconstruction like human cognitive systems.

[0003] In existing technologies, the Transformer model and its variants have been widely applied to sequence modeling and information processing tasks. For example, the standard Transformer model utilizes a self-attention mechanism to process sequence data, the Set Transformer optimizes ensemble data processing through Induced Attention Blocks (ISAB), and the Shared Workspace Transformer (TR+HSW) employs an inter-layer shared workspace strategy and a Top-k competition mechanism to achieve efficient information storage. However, these methods primarily rely on direct gradient optimization or attention mechanisms to update working memory, lacking robustness against noise and interference, and failing to simulate the reflective process in human cognition—that is, self-examination, adjustment, and correction of existing knowledge to adapt to new information and environments. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a visual reasoning solution that improves adaptability in complex environments by making full use of working memory, which is responsible for temporarily storing and processing information, and long-term memory, which stores stable knowledge representations, through effective interaction between working memory and long-term memory.

[0005] The technical solution adopted by this invention to solve the above-mentioned technical problems is a visual reasoning method based on reflective working memory, comprising:

[0006] Visual feature analysis steps: Perform target detection and feature extraction on the input image to extract the visual features of the target;

[0007] Working memory encoding: Store the visual features of the target related to the reasoning task into the working memory module to establish a temporary information representation;

[0008] Reasoning steps: The working memory module adds noise to the temporary information representation through working memory and denoises guided by long-term memory to complete multiple rounds of memory-guided reasoning and dynamic memory updates, gradually obtaining a visually optimized information representation; the long-term memory is reusable prior cross-task general knowledge stored in the knowledge base retrieval or historical state cache; the working memory is the dynamic state related to the current visual reasoning task.

[0009] Result output steps: Integrate the visually optimized information representation in the working memory module to generate visual reasoning result output;

[0010] The process of adding noise to working memory and denoising guided by long-term memory is as follows:

[0011] A forward noise-adding operation is applied to the working memory to simulate noise interference in the working memory. The noise-adding working memory is then concatenated with a weighted long-term memory to form an intermediate state of the working memory, so that the intermediate state of the working memory contains effective guidance information from the long-term memory for the noise-adding working memory. The intermediate state of the working memory is then input into a flow matching network with an attention mechanism. The output of the flow matching network is used to remove noise components from the working memory to simulate the denoising process of the working memory, thereby obtaining the restored noise-free working memory as a representation of video optimization information.

[0012] This invention achieves working memory optimization based on a reflective mechanism by adding noise to working memory and guiding denoising from long-term memory. This enables effective interaction between working memory and long-term memory, allowing working memory to self-adjust and correct itself using stable knowledge from long-term memory. This enhances the system's long-term stability and information retention, maintaining stable memory representations in visual reasoning tasks, making it less susceptible to interference, and strengthening its self-regulation capabilities during visual reasoning. A flow matching network is introduced to implement the denoising process, constructing the working memory optimization process as a continuous-time dynamic system. This allows working memory to gradually recover to a noise-free state along the optimal trajectory, enhancing the robustness and adaptability of memory representations.

[0013] The beneficial effects of this invention are: it effectively simulates the reflective process in human cognition, enabling working memory to adaptively adjust and optimize under the guidance of long-term memory, thus improving the performance of visual reasoning tasks in dynamic and complex environments. The introduction of a flow matching network to achieve the denoising process significantly improves the model's ability to recognize geometric structures in images, especially in low-resource training environments. Furthermore, the entire memory optimization process can still run efficiently even in environments with limited computing resources, exhibiting high computational efficiency and moderate training and inference overhead. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the denoising process of the flow matching network in an embodiment of the present invention. Detailed Implementation

[0015] The visual reasoning method of the embodiment includes: a visual feature parsing step, a working memory encoding step, a reasoning step, and a result output step.

[0016] In the reasoning process, a reflection module is constructed by combining long-term memory and working memory through noise addition and denoising mechanisms. This module applies noise perturbation to working memory to change its state and then uses long-term memory for denoising, thereby forming an adaptive memory update process.

[0017] Reflective working memory optimization includes two main processes: adding noise to working memory and guiding denoising in long-term memory. These are illustrated in detail below with reference to the accompanying diagram:

[0018] The process of adding noise to working memory includes:

[0019] S1, Noise Addition to Working Memory:

[0020] S11. Obtain the working memory representation updated by the contention write and forgetting mechanisms. Where R is the real number field, N represents the number of working memory cells, and D... m Its feature dimension is t, which represents the time step of the original working memory; in the understanding task, N=8, D m The value can be set to 128-256 depending on the characteristics of the task; for example, it can be set to 128 in a geometric vision task.

[0021] S12. Apply forward noise addition to the working memory to simulate noise interference in the working memory and enhance the robustness of the model:

[0022]

[0023] in, This represents the working memory after applying noise, where Z is standard Gaussian noise, and t ′ This represents the time step of the denoising process in the reflection module, controlling the noise level. When t... ′ When = 0, Remain in its original state; when t ′ When = 1, It is completely converted into noise. This process ensures that working memory is perturbed by noise to varying degrees at different time steps, thereby simulating the gradual degradation of information during visual cognition.

[0024] The long-term memory-guided denoising process includes:

[0025] S2, Denoising Flow Field Modeling:

[0026] S21. Construct a Stream Matching Network (FMN) for predicting and denoising vector fields:

[0027]

[0028] in, For noisy current working memory, The long-term memory, updated after the outer product operation, serves as stable background information to guide denoising. The number of long-term memory segments M is set to 5, and the size of each long-term memory segment is 5×D. m The neural network v(·) aims to adjust... To approach its original, noise-free state

[0029] S3, Training of the Stream Matching Network:

[0030] S31. To ensure training stability and convergence, the Adam optimizer is used, with the learning rate set between 0.0001 and 0.0002 (e.g., 0.0002 in the bAbI task, and 0.0001 in the Sort-of-CLEVR and Triangle tasks). The batch size is 64-100, and Dropout with a regularization of 0.1 is applied to prevent overfitting.

[0031] S32. Learning from denoised flow fields is key to achieving memory retrieval. Working memory. The evolution is described by the following formula:

[0032]

[0033] This equation shows that the rate of change of working memory is affected by its current noise disturbance. The decision is made. Therefore, the goal of training is to enable the neural network to learn a vector field v that serves as a noise component. θ This brings it closer to the actual evolutionary trajectory:

[0034]

[0035] S33. Therefore, a loss function is adopted. Optimize:

[0036]

[0037] E represents the expectation.

[0038] The optimization objective of this loss function is to make v θ Estimating noise components This will restore noise-free working memory.

[0039] S4. Noise reduction process:

[0040] S41. In the denoising process, the recovery process of working memory is modeled using ordinary differential equations (ODEs). Specifically, working memory... The evolution over time can be described by the following equation:

[0041]

[0042] in, This is a denoised flow field, learned through a neural network. The negative sign indicates that the flow field is corrected in the direction of reducing noise in the working memory.

[0043] S42. Set the working memory at the initial time t ′ =1 is the state This refers to working memory that is completely contaminated with noise. This extreme noise setting helps enhance the model's resistance to interference, especially making it more robust in complex visual reasoning tasks such as Sort-of-CLEVR and bAbI.

[0044] S43. Use numerical methods (such as the Runge-Kutta method) to obtain data from t. ′ =1, start by solving in reverse, gradually approximating t ′ Noise-free working memory when =0. Depending on the complexity of different tasks, the number of model layers is set to 2-8. For example, a 2-layer structure is used in the Triangle task, while an 8-layer structure is used in the bAbI task, to adapt to the inference complexity of different tasks.

[0045] S44. After the noise-reducing flow field is corrected, the noise-free working memory state is restored. This refers to the final memory representation after noise removal. The initial gating factor is set to 0.7-0.75 to balance the needs of memory retention and updating.

[0046] See Figure 1 The flow matching network architecture includes:

[0047] Information interaction module A:

[0048] To fully utilize long-term memory to guide denoising, the flow matching network employs an attention mechanism to enhance the working memory of noisy processes. With long-term memory To enable interaction, specifically, the query vector, key, and value are first obtained through linear projection:

[0049]

[0050] Among them, W q W k W v This is a learnable projection matrix. The number of attention heads is set according to the task complexity, set to 8 in text reasoning tasks (such as bAbI), and reduced to 4 or 1 in image tasks.

[0051] Subsequently, the attention weight α is calculated to obtain the weighted long-term memory representation:

[0052]

[0053] in, D represents information that enhances working memory from long-term memory. k Let K be the dimension of the key, and T be the transpose. To limit the scope of attention calculations and improve computational efficiency, Top-k is set to 5.

[0054] B-level noise reduction calculation module:

[0055] Noisy working memory With weighted long-term memory splicing together to form a new working memory intermediate state M int :

[0056]

[0057] This intermediate state contains effective guidance information from long-term memory for noisy working memory.

[0058] Next, the intermediate state is concatenated with the time information t′ expanded to the same dimension and fed into a multilayer perceptron for processing. This perceptron contains two linear transformation layers, each followed by a ReLU activation function to enhance the network's non-linear expressive power. Four to five MLP layers are used in the attention module to ensure sufficient expressive power.

[0059] Finally, a linear transformation is used to map the hidden layer output to the dimension of the denoising vector field, resulting in the network output:

[0060]

[0061] The output vector field is used to remove noise components from working memory, thereby restoring noise-free working memory. By introducing the guiding role of long-term memory and dynamic adjustment of temporal information, the flow matching network can accurately simulate the denoising process of working memory, achieving more stable and flexible cognitive regulation in complex environments.

[0062] Model evaluation:

[0063] In understanding tasks (such as Sort-of-CLEVR, bAbI, and Equilateral Triangle), accuracy is used as the primary evaluation metric.

[0064]

[0065] This metric intuitively reflects the model's visual reasoning ability. In the experiments, all tasks used a random seed of 1 to ensure the reproducibility of the results.

[0066] All experiments were conducted in a high-performance computing environment equipped with an NVIDIA V100 GPU (16GB or 24GB of VRAM) to ensure the stability and efficiency of training and inference. For visual inference tasks (such as Sort-of-CLEVR and Equilateral Triangle), model training for 200 epochs takes approximately 8 hours.

Claims

1. A visual reasoning method based on reflective working memory, characterized in that, include: Visual feature analysis steps: Perform target detection and feature extraction on the input image to extract the visual features of the target; Working memory encoding: Store the target visual features relevant to the reasoning task into the working memory module to output the initial working memory; Reasoning steps: The working memory module based on the flow matching network adds noise to the working memory and guides denoising through long-term memory to complete multiple rounds of memory-guided reasoning and dynamic memory updates, gradually obtaining a visually optimized information representation; the long-term memory is reusable prior cross-task general knowledge stored in the knowledge base retrieval or historical state cache; the working memory is the dynamic state related to the current visual reasoning task. Output results: Integrate the visually optimized information representations in the working memory module to generate visual reasoning results output; The process of adding noise to working memory and denoising guided by long-term memory is as follows: A forward noise-adding operation is applied to the working memory to simulate noise interference in the working memory. The stream matching network concatenates the noise-adding working memory with the weighted long-term memory to form an intermediate state of the working memory, so that the intermediate state of the working memory contains effective guidance information from the long-term memory for the noise-adding working memory. The output of the stream matching network is used to remove the noise components in the working memory to simulate the denoising process of the working memory, thereby obtaining the restored noise-free working memory as a representation of video optimization information.

2. The method as described in claim 1, characterized in that, The specific process of adding noise to working memory is as follows: A1. Representing working memory as Where t represents the time step of working memory; A2. Apply a forward noise-adding operation to the working memory to obtain the noise-adding working memory. Where Z is standard Gaussian noise, and t′ represents the time step of the denoising process, used to control the noise level, with a value ranging from 0 to 1.

3. The method as described in claim 2, characterized in that, The specific process of long-term memory guidance is as follows: Represent long-term memory as Set the initial denoising process time step t′ = 1, and the working memory after denoising... The initial state is That is, working memory that is completely contaminated by noise; Representing long-term memory Noisy working memory The time step t′ of the denoising process is input into the flow matching network (FMN). The flow matching network obtains the attention weights of long-term memory through the attention mechanism, concatenates the noisy working memory with the weighted long-term memory to form a working memory intermediate state, and then concatenates the working memory intermediate state with the time step t′ of the denoising process extended to the same dimension and inputs it into a multilayer perceptron for processing. The multilayer perceptron contains two linear transformation layers, each followed by a ReLU activation function to enhance the nonlinear expressive power of the network. The multilayer perceptron outputs the predicted noise component. The noise reduction process of working memory is achieved by solving ordinary differential equations based on the predicted noise components. Numerical methods are used to solve the equations in reverse from t′=1, gradually approximating the noise-free working memory at t′=0, until the noise-free working memory is restored.

4. The method as described in claim 3, characterized in that, The specific implementation of the stream matching network to obtain the attention weights for long-term memory through the attention mechanism is as follows: Representation of long-term memory Noisy working memory The query vector Q, key K, and value V are obtained through linear projection: Among them, W q W k W v These are the learnable projection matrices for the query vector Q, key K, and value V, respectively. Calculate the attention weight α: Among them, D k Let K be the dimension of the key, and T be the transpose. We use weighted long-term memory, represented as αV.

5. The method as described in claim 4, characterized in that, When weighting long-term memories, a Top-k sparse attention mechanism is used.

Images