A visual method for brain-like target recognition inference chain

Abstract

The present application relates to a kind of brain-like target identification inference chain visualization method, belong to image recognition technical field, solve the problem of lack of visualization presentation in the inference intermediate process in prior art. Method includes: obtaining trained brain-like target identification model;Brain-like target identification model includes multiple low-level experts LoRA and one total expert LoRA;The image to be identified and instruction are input into the brain-like target identification model;In the process that brain-like target identification model reasons input image, the intermediate state data generated inside brain-like target identification model is captured;Intermediate state data includes: cognitive anchor point sequence, the activation weight of each time step different network layer on each low-level expert LoRA and the interaction weight between each expert LoRA;Based on intermediate state data, visualization graphics is generated, visualization image includes expert interaction directed graph and / or expert activation intensity space-time heat map. The visualization presentation of model internal inference chain is realized.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a visualization method for a brain-like target recognition reasoning chain. Background Technology

[0002] In recent years, visual target recognition technology based on deep neural networks has made significant progress and has been widely used in scenarios such as autonomous driving, security monitoring, and robot navigation. However, traditional target recognition models generally adopt an end-to-end black-box architecture, and their internal decision-making processes are highly nonlinear and difficult to analyze.

[0003] The lack of effective means to visualize the inference process of target recognition models results in the interpretability of the models remaining at a superficial level of knowing what the model outputs, rather than achieving a deeper understanding of why the model outputs in that way. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a visualization method for brain-like target recognition reasoning chain, in order to solve the problem of the lack of visualization of the intermediate reasoning process in existing methods.

[0005] On one hand, embodiments of the present invention provide a visualization method for a brain-like target recognition reasoning chain, comprising the following steps: A trained brain-like target recognition model is obtained; the brain-like target recognition model includes multiple low-level expert LoRAs and a total expert LoRA; each low-level expert LoRA corresponds to a low-level visual cognitive task, and the total expert LoRA is used to activate the low-level expert LoRAs and perform feature interaction; The image to be recognized and the instruction are input into the brain-like target recognition model; during the reasoning process of the brain-like target recognition model on the input image, the intermediate state data generated inside the brain-like target recognition model is captured; the intermediate state data includes: cognitive anchor sequence, activation weights of each low-level expert LoRA on different network layers at each time step, and interaction weights between each expert LoRA. A visualization is generated based on the intermediate state data, and the visualization includes a directed graph of expert interaction and / or a spatiotemporal heatmap of expert activation intensity.

[0006] Based on a further improvement of the above method, the expert interaction directed graph is used to show the information flow direction and interaction intensity among all activated experts when generating each anchor point. The nodes in the directed graph of expert interaction represent active expert LoRAs, the size or color of the node represents the interaction influence of the expert LoRA, the directed edges represent the information interaction between active expert LoRAs, and the width or color of the edge represents the total interaction weight between expert LoRAs.

[0007] Based on the above method, the total interaction weights among experts in LoRA are obtained in the following way:

[0008] Where L represents the number of layers in the brain-like target recognition model, This indicates that at time step t, the distance from expert i's LoRA to expert j's LoRA is at time step t. Layer interaction weights Represents the t-th time step. A set of expert LoRA sets for layer activation. This indicates the number of tokens contained in the k-th anchor. This represents the set of time steps for generating the k-th anchor point. Indicates an indicator function, Indicates a relationship.

[0009] Based on a further improvement of the above method, the spatiotemporal heatmap of expert activation intensity is used to display the activation intensity of each expert LoRA at different time steps and / or different network layers. The spatiotemporal heatmap of expert activation intensity includes the anchor-expert heatmap and the network layer-expert heatmap; The anchor-expert heatmap is a two-dimensional matrix diagram, with the vertical axis representing the LoRA (Lower Expert Registry) and the horizontal axis representing the anchor points; the elements in the matrix... This represents the activation strength of the i-th low-level expert LoRA when generating the k-th anchor point; The network layer-expert heatmap is a two-dimensional matrix, with the vertical axis representing the low-level expert LoRA and the horizontal axis representing the network layers of the brain-like target recognition model. The elements in the matrix... This indicates that the i-th low-level expert LoRA is at the . The activation intensity of the layer.

[0010] A further improvement to the above method is characterized by obtaining the trained brain-like target recognition model in the following manner: Multiple low-level expert LoRAs are introduced on the pre-trained base model. Each low-level expert LoRA corresponds to a low-level visual cognition task. The corresponding low-level expert LoRA is trained based on the training dataset corresponding to each low-level visual cognition task. A total expert LoRA is introduced on the pre-trained base model; the total expert LoRA is used to extract latent features, and low-level expert LoRAs are dynamically activated based on the latent features. Enhanced features are generated through feature interaction based on the activated low-level expert LoRAs. Construct a process-supervised training dataset; train the overall expert LoRA on the process-supervised training dataset; and obtain a trained brain-like target recognition model.

[0011] Further improvements to the above method involve dynamically activating low-level expert LoRA based on latent features, and then generating enhanced features through feature interaction based on the activated low-level expert LoRA, including: For each layer of the pre-trained base model: Whether to activate a low-level expert LoRA in a layer is determined based on the similarity between the latent features of the total expert LoRA and each low-level expert LoRA in that layer. Enhanced features for this layer are generated by feature interaction between the latent features of the activated low-level expert LoRA and the total expert LoRA.

[0012] Further improvements to the above method involve generating enhanced features through feature interaction between the latent features of the activated low-level expert LoRA and the total expert LoRA, including: Based on the activated expert LoRA, the latent features of the activated expert LoRA are interacted with to obtain interactive features through the up-dimensional matrix of this layer; The interaction weights of the interaction features are obtained by dynamically assigning weights to the interaction features. The enhanced features of this layer are obtained by adding the output features of the pre-trained base model at this layer to the interaction features weighted by interaction weights.

[0013] Based on the further improvement of the above method, interactive features are obtained by interacting the latent features of the activated expert LoRA with the increased dimension matrix of this layer, including: Each activated expert LoRA is designated as the source expert, and each other activated expert LoRA (excluding the source expert itself) is designated as the target expert. The latent features of the source experts are transformed using a trainable interaction matrix; The transformed features are projected through the target expert's upscaling matrix to obtain the interaction features corresponding to the source expert and the target expert.

[0014] Based on the above method, a further improvement is made to dynamically assign weights to the interaction features to obtain the interaction weights of the interaction features, including: The latent features of the overall expert LoRA are used as the query vector; For each interaction feature, calculate the key vector using the up-dimensional matrix of the source expert corresponding to that interaction feature; The interaction weight of this interaction feature is calculated based on the query vector and key vector.

[0015] Based on the further improvement of the above method, each low-level expert LoRA includes multiple sets of trainable low-rank matrix pairs. Each set of trainable low-rank matrix pairs is set in each layer of the pre-trained base model and is used to perform low-rank updates on the original weights in each layer of the base model.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. This invention captures the cognitive anchor sequence, expert interaction weights, and activation intensity data generated by a brain-like target recognition model during the reasoning process, and generates a directed graph of expert interactions and a spatiotemporal heatmap of activation intensity. This presents the reasoning chain, originally hidden within the model, in an intuitive graphical way. Users can clearly see the types of experts the model relies on when generating each cognitive anchor, the direction and intensity of information flow between experts, and the dynamic changes in expert activation intensity in different reasoning steps. This overcomes the limitation of traditional black-box models that only provide results without explanation, making the model's decision-making basis traceable and verifiable.

[0017] 2. In high-risk application scenarios such as autonomous driving and medical imaging, users need to verify whether the model's judgment criteria are reasonable. The visualization provided by this invention allows users to examine the visual attributes that the model focuses on when identifying targets, enabling users to make decisions on whether to adopt the model's results based on an understandable reasoning chain, thereby establishing human-machine mutual trust.

[0018] 3. By distributively pre-training each low-level expert LoRA, each low-level expert LoRA is responsible for a specific function, similar to the brain partitioning to perform a task at a certain cognitive level. The feature extraction of low-level expert LoRA is decoupled from the final target category, so that the model can still rely on the stable perception of low-level attributes such as color and direction to make inferences in different domains, without over-relying on the statistical patterns of specific domains.

[0019] 4. Based on the pre-trained base model and the trained low-level expert LoRA, a total expert LoRA is introduced. During the prediction process, the model dynamically activates a portion of the low-level experts LoRA related to the current task, enabling them to fully integrate and interact. This simulates the top-down modulation and parallel processing of the human brain, exhibiting a working mechanism similar to the brain's regional collaboration and cognitive hierarchical interaction. This approach aligns with the human brain's cognitive process, providing interpretability while fully utilizing the reasoning ability of the pre-trained model to improve the model's recognition accuracy.

[0020] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0021] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart illustrating a visualization method for a brain-like target recognition reasoning chain according to an embodiment of the present invention. Detailed Implementation

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0023] A specific embodiment of the present invention discloses a visualization method for brain-like target recognition reasoning chains, such as... Figure 1 As shown, it includes the following steps: S1. Obtain a trained brain-like target recognition model; the brain-like target recognition model includes multiple low-level expert LoRAs and a total expert LoRA; each low-level expert LoRA corresponds to a low-level visual cognitive task, and the total expert LoRA is used to activate the low-level expert LoRAs and perform feature interaction; S2. Input the image to be recognized and the instruction into the brain-like target recognition model; during the reasoning process of the brain-like target recognition model on the input image, capture the intermediate state data generated inside the brain-like target recognition model; the intermediate state data includes: cognitive anchor sequence, activation weights of each low-level expert LoRA on different network layers at each time step, and interaction weights between each expert LoRA. S3. Generate a visualization based on the intermediate state data. The visualization includes a directed graph of expert interaction and / or a spatiotemporal heatmap of expert activation intensity.

[0024] Compared with existing technologies, the visualization method for the brain-like target recognition reasoning chain provided in this embodiment obtains a trained brain-like target recognition model, inputs the image to be recognized and the instructions into the model, and captures the cognitive anchor sequence, expert interaction weights and activation intensity data generated within the model during the reasoning process. This generates a directed graph of expert interactions and a spatiotemporal heatmap of activation intensity, presenting the reasoning chain, originally hidden within the model, in an intuitive graphical way. Users can clearly see the type of expert the model relies on when generating each cognitive anchor, the direction and intensity of information flow between experts, and the dynamic changes in expert activation intensity in different reasoning steps. This breaks through the limitation of traditional black-box models that "only provide results, not reasons," making the model's decision-making basis traceable and verifiable.

[0025] Human cognitive processes unfold in a progressive manner. At the lowest level, the brain distinguishes visual attributes such as local contrast, direction, color, and motion. The intermediate level involves analyzing scene layout and surface attributes, resolving visual images into surfaces and overall contours, and distinguishing foreground from background. The highest level involves target recognition. A trained brain-like target recognition model simulates the working mechanism of brain regional collaboration and cognitive level interaction. Each low-level expert LoRA is responsible for a specific function, similar to a brain region performing a task at a certain cognitive level. The feature extraction of low-level expert LoRAs is decoupled from the final target category, allowing the model to reason based on stable perception of low-level attributes such as color and direction in different domains (e.g., changes in lighting, differences in scene style), without overly relying on domain-specific statistical patterns. During prediction, the overall expert LoRA dynamically activates a portion of low-level expert LoRAs relevant to the current task, enabling them to fully integrate and interact, simulating the top-down modulation and parallel processing of the human brain. This aligns with the human cognitive process, providing interpretability while fully utilizing the reasoning ability of the pre-trained model, thus improving the model's recognition accuracy.

[0026] Specifically, the trained brain-like target recognition model is obtained using the following method: S11. Introduce multiple low-level expert LoRAs on the pre-trained base model. Each low-level expert LoRA corresponds to a low-level visual cognition task. Train the corresponding low-level expert LoRA based on the training dataset corresponding to each low-level visual cognition task. S12. Introduce a total expert LoRA on the pre-trained base model; the total expert LoRA is used to extract latent features, dynamically activate low-level expert LoRAs based on the latent features, and generate enhanced features through feature interaction based on the activated low-level expert LoRAs. S13. Construct a process-supervised training dataset; train the overall expert LoRA on the process-supervised training dataset; obtain a trained brain-like target recognition model.

[0027] In implementation, the pre-trained base model can employ existing multimodal large models, such as Qwen2-VL-2B and GPT, which support multimodal information such as images and text as input. These multimodal large models include an image encoder, a visual adapter, and a large language model. The image encoder extracts image features, the visual adapter maps image features to a feature space consistent with text features, and the large language model acts as a decoder to understand features, perform logical reasoning, and output results. Large language models typically employ an autoregressive approach for generation, predicting the next token based on previous tokens in the generated sequence. This predicted token is then concatenated into the sequence as input for the next iteration. In other words, when predicting each token using an autoregressive approach, it utilizes not only the feature information of the original input image and text but also the features of already generated tokens. This is existing technology and will not be elaborated upon in this invention. This invention does not impose any limitations on the multimodal large model.

[0028] In this embodiment of the invention, the parameters of the pre-trained base model are always frozen.

[0029] Multiple low-level expert LoRAs are introduced on the pre-trained base model, with each low-level expert LoRA corresponding to a low-level visual cognitive task (such as color recognition, orientation judgment, edge detection, etc.).

[0030] In implementation, each low-level expert LoRA includes multiple sets of trainable low-rank matrix pairs. Each set of trainable low-rank matrix pairs is set in each layer of the pre-trained base model and is used to perform low-rank updates on the original weights in each layer of the base model.

[0031] In implementation, a low-level expert LoRA is added to the decoder of the pre-trained base model (large language models are typically composed of multiple stacked decoders). For example, the decoder of the pre-trained base model has a total of Layer (usually a Transformer layer), the first The original weight matrix of a layer (e.g., the query projection of the attention layer) is denoted as The original weight matrix is ​​frozen, where Represent real numbers, and This indicates the dimension of the original weight matrix.

[0032] For the i-th low-level expert LoRA, its corresponding multiple pairs of low-rank matrices are denoted as follows: . ( This indicates that the i-th low-level expert LoRA is at the . The low-rank matrix pairs corresponding to the layers, Represents a dimension reduction matrix. Let r represent the higher-dimensional matrix, and r represent the rank of the lower-rank matrix.

[0033] During the forward propagation process of training the i-th low-level expert LoRA, the i-th The actual weight matrix of the layer is: .

[0034] The updated actual weight matrix is ​​applied to the input features (multiplied by the input features) to extract feature information for the low-level visual cognitive task corresponding to the i-th low-level expert LoRA, generating cognitive anchor features, which are then input to the next layer. The cognitive anchor features extracted in the last layer are input to the output head of the pre-trained base model, and the output head autoregressively generates the next token in the response sequence.

[0035] In implementation, the trainable low-rank matrices of each low-level expert LoRA are set up side-by-side, and each low-level expert LoRA is trained separately. That is, for the training of the i-th low-level expert LoRA, the trained model is the pre-trained base model + Since each low-level expert LoRA corresponds to a different low-level visual cognition task, it is necessary to construct a training dataset for each low-level expert LoRA.

[0036] For example, for the i-th low-level visual cognition task, its training dataset is:

[0037] in, Represents the training dataset The number of samples in This represents the image to be identified for the j-th sample. The instruction representing the j-th sample, This represents the expected response text for the j-th sample.

[0038] For example, the i-th low-level expert LoRA is a color perception task, and the sample instruction is "determine the dominant color of the image", with the expected response text, such as "color: red".

[0039] Train the low-level expert LoRA on the training dataset corresponding to the i-th low-level expert LoRA, and update the low-rank matrix pair of the low-level expert LoRA.

[0040] In implementation, the loss function for low-level expert LoRA is the cross-entropy loss modeled by autoregressive language:

[0041]

[0042] in, This represents the training dataset corresponding to the i-th low-level expert LoRA. The number of samples, This represents the token length of the response to the j-th sample. Represents the first to the last part of the answer of the j-th sample. A real token, This represents the response of the j-th sample. A real token, Indicates in ( Under the condition that the model predicts the j-th sample, the first j-th sample is predicted. Each token is The probability, This represents the currently trainable parameters, i.e., the low-rank matrix pair of the i-th low-level expert LoRA.

[0043] The training objective is to minimize Since the pre-trained base model is frozen, during training, gradient descent is used to update only the low-rank matrix pair of the i-th low-level expert LoRA. .

[0044] Training stops when the loss function converges, and a set of parameters is obtained. That is, the i-th low-level expert LoRA.

[0045] After training each low-level expert LoRA, freeze the parameters of each low-level expert LoRA. .

[0046] Each expert focuses on an extremely simple cognitive atomic task, requiring little data (only a few hundred images per class), enabling fast training, and the experts are completely decoupled from each other. In subsequent centralized learning phases, these atomic experts are flexibly combined to complete complex target recognition tasks.

[0047] The decoupling of low-level expert feature extraction from the final target category allows the model to reason based on stable perceptions of low-level attributes such as color and direction across different domains (e.g., variations in lighting and scene style), without over-reliance on domain-specific statistical patterns. Furthermore, the model learns cross-task-compatible basic visual features (color, direction, etc.), which can be reused in different object detection tasks. Especially when training samples for the target category are scarce, the rich features provided by low-level experts can compensate for the lack of high-level semantic information.

[0048] To coordinate and utilize low-level expert LoRA, a total expert LoRA is introduced on the pre-trained base model.

[0049] Specifically, the total expert LoRA consists of multiple sets of trainable low-rank matrix pairs, each set of trainable low-rank matrix pairs is set in each layer of the pre-trained base model; the trainable low-rank matrix pairs of the total expert LoRA are set side by side with the low-rank matrix pairs of each low-level expert LoRA.

[0050] During implementation, the multiple low-rank matrix pairs corresponding to the overall expert LoRA are denoted as follows: . ( Chief Expert LoRA stated at the The low-rank matrix pairs corresponding to the layers, Represents a dimension reduction matrix. Let r represent the higher-dimensional matrix, and r represent the rank of the lower-rank matrix.

[0051] When training the overall expert LoRA, the parameters of each lower-level expert LoRA are frozen.

[0052] Specifically, the chief expert LoRA uses the following methods to extract latent features:

[0053] in, This indicates that the pre-trained basis model is in the first... The input features of the layer Chief Expert LoRA stated at the Latent characteristics of the layer.

[0054] This involves dynamically activating low-level expert LoRA based on latent features, and then generating enhanced features through feature interaction based on the activated low-level expert LoRA. Specifically, this includes: For each layer of the pre-trained base model: S121. Based on the similarity between the latent features of the total expert LoRA and each low-level expert LoRA at this layer, determine whether to activate the low-level expert LoRA at this layer; S122. Generate enhanced features for this layer by performing feature interactions between the latent features of the activated low-level expert LoRA and the total expert LoRA.

[0055] By dynamically activating the lower-level experts (RoLA) most relevant to the current task based on the overall expert (LoRA), interference from irrelevant lower-level experts is avoided, allowing high-level reasoning to focus on the most useful information.

[0056] Specifically, the i-th low-level expert LoRA in the 1st... The latent characteristics of the layer are:

[0057] in, This indicates that the pre-trained basis model is in the first... The input features of the layer This indicates that the i-th low-level expert LoRA is at the . Latent characteristics of the layer.

[0058] During implementation, the cosine similarity function is used to calculate the latent features of the overall expert LoRA at this layer. Latent features of the i-th low-level expert LoRA The similarity between the parameters is calculated. If the similarity exceeds a preset similarity threshold, the i-th low-level expert LoRA is activated and allowed to participate in subsequent calculations; otherwise, the low-level expert LoRA is skipped. In implementation, the similarity threshold can be set according to the specific task similarity requirements, for example, to 0.6.

[0059] It should be noted that the overall expert LoRA is always active.

[0060] It should be noted that the same low-level expert LoRA may be in different activation states at different layers—activated at one layer and not activated at another. For example, the color expert LoRA may be activated at shallow layers because the input features contain strong raw color information, and the overall expert LoRA has a high similarity to it; at deeper layers, the input features have been abstracted into high-level semantic features, and the similarity with color features decreases, so the color expert LoRA may no longer be activated.

[0061] This layered, dynamic activation pattern simulates the phenomenon that the degree of participation of different visual regions in the human brain in different cognitive tasks varies with the depth of processing.

[0062] Then, feature interactions are performed between the latent features of the activated low-level expert LoRA and the total expert LoRA to generate enhanced features for this layer. Specifically, this includes: S1221. Based on the activation of the expert LoRA, the latent features of the activated expert LoRA are interacted with by the up-dimensional matrix of this layer to obtain interactive features. S1222. Dynamically assign weights to interactive features to obtain their interactive weights. S1223. The output features of the pre-trained base model at this layer are added to the interaction features weighted by interaction weights to obtain the enhanced features of this layer.

[0063] During implementation, the first The expert LoRA activated at each layer includes the total expert LoRA and the expert LoRA at the second layer. The low-level expert LoRA is activated in the layer.

[0064] Record No. The expert LoRA set for layer activation is Including the chief expert LoRA and in the 19th The low-level expert LoRA is activated in the layer.

[0065] Based on the activated expert LoRA, the latent features of the activated expert LoRA are interacted with at this layer using the increased-dimensional matrix to obtain interactive features, including: S12211. Take each activated expert LoRA as the source expert, and take each other activated expert LoRA other than the source expert itself as the target expert. S12212, The latent features of the source experts are transformed using a trainable interaction matrix; S12213. Project the transformed features through the target expert's upscaling matrix to obtain the interaction features corresponding to the source expert and the target expert.

[0066] During implementation, for each pair of source experts and target experts, perform the following operations: The latent features of the source experts are transformed using a trainable interaction matrix:

[0067] In the set of activated expert LoRAs, the i-th expert LoRA is the source expert, and the j-th expert LoRA is the target expert. Source experts indicated in the first Latent characteristics of the layer Let LoRA of expert i and LoRA of expert j be the same as LoRA of expert i. The trainable interaction matrix of the layer. .

[0068] Transformed features Projecting back to the original feature space using the target expert dimension-upgrading matrix:

[0069] in, Indicates that the target expert is in the first The dimension-upgrading matrix of the layer.

[0070] For sets Each pair of source and target experts in the algorithm calculates interaction features to obtain an interaction feature set. .

[0071] Then, dynamic weights are assigned to the interaction features to obtain their interaction weights. Specifically, this includes: The latent features of the overall expert LoRA are used as the query vector; For each interaction feature, calculate the key vector using the up-dimensional matrix of the source expert corresponding to that interaction feature; The attention score of each interaction feature is calculated based on the query vector and key vector to obtain the interaction weight of each interaction feature.

[0072] During implementation, the latent features of the overall expert LoRA are used as the query vector. .

[0073] For each interaction feature, calculate its key vector. ,in, Representing interaction features The corresponding source expert is in the first The layer is an upgraded matrix, with the superscript T indicating transpose.

[0074] The interaction weight of each interaction feature is obtained by calculating the attention score of the interaction feature based on the query vector and key vector:

[0075]

[0076] in, This indicates that the source experts and target experts have a view on ( The interaction weights corresponding to the interaction features of ) This indicates that the source experts and target experts have a view on ( Attention scores corresponding to the interaction features of ) The set of expert LoRAs that are activated Source experts and target experts in the process, This indicates that the source experts and target experts have... Attention scores corresponding to the interaction features.

[0077] The pre-trained base model in the... The layer's output features are added to the interaction features weighted by interaction weights to obtain the layer's enhanced features:

[0078] in, This indicates that the pre-trained basis model is in the first... The input features of the layer The set of activated expert LoRAs is: Source experts and target experts in the process enhance features As the first The output of one layer is then passed on to the next layer.

[0079] By dynamically fusing and activating expert outputs through an attention mechanism, the contributions of different cognitive channels can be adjusted according to the actual situation of the input image, thereby improving robustness to domain shifts and enhancing the model's generalization and cross-domain adaptability.

[0080] By dynamically activating low-level experts associated with current local features (e.g., when the target is partially occluded, it may rely more on color and edge information than on complete part information), the model can still make reasonable inferences even with incomplete information. Simultaneously, the outputs of multiple low-level experts are fused through attention, effectively achieving soft selection of multi-view features and exhibiting better robustness to scale changes and partial occlusion.

[0081] The features output from the final layer are fed into the output head of the pre-trained base model, which then autoregressively generates the next token in the response sequence.

[0082] During implementation, the LoRA algorithm is trained on the process-supervised training dataset to obtain a well-trained brain-like target recognition model.

[0083] In the process-supervised training dataset of this invention, each sample includes three parts: image, instruction, and response.

[0084] use This represents the process-supervised training dataset. Wherein, Represents the process of supervised training dataset The number of samples in This represents the image to be identified for the i-th sample. The instruction representing the i-th sample. This represents the expected response text for the i-th sample.

[0085] The instructions explicitly require the model to generate low-level visual anchors before the final answer, such as "Please identify the target in the image and describe its color, direction, and edge features in sequence during the thinking process, and finally output the target category".

[0086] The answer is an autoregressive sequence of target text, composed of multiple segments connected sequentially:

[0087] This represents the k-th low-level visual perception anchor point. Each anchor point is a natural language description, such as "color: red", "direction: horizontal", and "edge: clear". This indicates the final target recognition result.

[0088] During implementation, images and instructions serve as input data for the pre-trained base model.

[0089] Because the training data explicitly includes low-level visual cognitive anchors (such as color, direction, and edges), and the model is forced to generate these anchors before outputting the final detection result during inference, the model's decision-making process is no longer a "black box," but rather forms a traceable and understandable step-by-step inference chain. Users can observe the anchor sequence generated by the model to determine whether its judgment criteria are correct, facilitating debugging and trust building.

[0090] In practice, the training loss also uses the cross-entropy loss from autoregressive language modeling:

[0091]

[0092] in, The sample set of the current training batch express The number of samples, This represents the answer of the j-th sample. The length of the token. Represents the first to the last part of the answer of the j-th sample. A real token, This represents the response of the j-th sample. A real token, Indicates in ( Under the condition that the model predicts the j-th sample, the first j-th sample is predicted. Each token is The probability, This represents the image of the j-th sample. The instruction representing the j-th sample, This represents the currently trainable parameters, including the low-rank matrix pairs of the total expert LoRA and the trainable interaction matrix.

[0093] The training objective is to minimize Updated via gradient descent Training stops when the training loss converges, resulting in a well-trained brain-like target recognition model. The well-trained brain-like target recognition model includes a pre-trained base model, multiple low-level expert LoRAs, and a total expert LoRA.

[0094] By inputting the image to be detected and the instructions into the trained brain-like target recognition model, the model can quickly and accurately output low-level cognitive anchors and the final detection result, forming a traceable and understandable step-by-step reasoning chain. Users can observe the anchor sequence generated by the model to determine whether its judgment basis is correct, which facilitates debugging and trust building.

[0095] To facilitate the explicit demonstration of the inference chain, showcasing the dynamic logic and causal chains within the model, intermediate state data is captured during the inference process based on the trained brain-like target recognition model and the image to be recognized. After inference, a visual image is generated based on the captured intermediate state data, thereby improving the model's interpretability and interactivity.

[0096] Specifically, the intermediate state data includes the cognitive anchor sequence, the activation weights of each low-level expert LoRA on different network layers at each time step, and the interaction weights between expert LoRAs.

[0097] It should be noted that in the autoregressive model, a token is generated at each time step. Here, a time step refers to the time unit corresponding to the generation of a token.

[0098] The activation weights of each low-level expert LoRA at each time step in different network layers, i.e., the similarity between the latent features of each low-level expert LoRA and the total expert LoRA at each layer at each time step, are denoted as: , indicating that at the t-th time step, the i-th low-level expert LoRA is at the t-th time step. The activation intensity of the layer.

[0099] The interaction weights among the LoRAs of each expert are the interaction weights calculated in step S1222. The foregoing explanation pertains to the process of generating a single token, without including time step information. At time step t, the source expert and the target expert... The interaction weights corresponding to the interaction features of ) are represented as .

[0100] The visualizations include expert interaction directed graphs and / or spatiotemporal heatmaps of expert activation intensity.

[0101] The expert interaction directed graph is used to show the direction of information flow and the intensity of interaction among all activated experts when generating each anchor point. This allows users to intuitively see how information is transferred between experts via LoRA. If the weight of a certain edge is abnormally high (e.g., color experts frequently send irrelevant information to direction experts), it may indicate training problems or data bias. By showing the expert interaction graph at each step, users can understand how the model gradually integrates multiple cognitive features before making a decision.

[0102] In implementation, the directed graph of expert interaction adopts a directed graph layout. The nodes of the directed graph of expert interaction represent active expert LoRAs, the size or color of the node represents the interaction influence of expert LoRAs, the directed edges represent the information interaction between active expert LoRAs; the width or color of the edge represents the total interaction weight between expert LoRAs.

[0103] In implementation, each expert interaction directed graph corresponds to an anchor point. By selecting different anchor points, the corresponding expert interaction directed graph is displayed. The nodes in the image represent the activated expert LoRAs (including the total expert LoRA and the activated low-level expert LoRAs). Directed edges represent the information transfer from one expert (source) to another expert (target). The thickness / color of the edges maps the interaction intensity (the higher the interaction intensity, the thicker the edge and the darker the color), thus even the information flow between experts when generating a certain feature anchor point.

[0104] In implementation, since one anchor point may correspond to multiple tokens, it is necessary to aggregate the interaction weights corresponding to multiple tokens to obtain the interaction weights between the activated experts when generating an anchor point.

[0105] Specifically, the total interaction weights among experts in LoRA are obtained using the following method:

[0106] Where L represents the number of layers in the brain-like target recognition model, This indicates that at time step t, the distance from expert i's LoRA to expert j's LoRA is at time step t. Layer interaction weights Represents the t-th time step. A set of expert LoRAs for layer activation. This indicates the number of tokens contained in the k-th anchor. This represents the set of time steps for generating the k-th anchor point. This indicates the indicator function, which is used when the i-th expert LoRA is in the set. And the j-th expert LoRA is also in the set. Output 1 if the output is 1, otherwise output 0. Indicates a relationship.

[0107] The expert-interactive directed graph is configured with interactive functionality. When a user clicks a node, the name of the corresponding expert's LoRA is displayed; when a user clicks an edge, the interaction weight value and the distribution at each layer are displayed. .

[0108] Through expert-interactive directed graphs, users can clearly track which lower-level experts' characteristics were passed to the head expert or other experts when the model generated each cognitive anchor point, thus understanding the "chain of evidence" upon which the final judgment relied. This breaks the limitation of traditional black-box models that only provide results without explanations, making the decision-making process traceable and auditable.

[0109] The spatiotemporal heatmap of expert activation intensity is used to show the changes in activation intensity of each expert LoRA at different time steps and / or different network layers.

[0110] During implementation, the spatiotemporal heatmap of expert activation intensity includes the anchor-expert heatmap and the network layer-expert heatmap.

[0111] During implementation, the anchor-expert heatmap is a two-dimensional matrix, with the vertical axis representing the LoRA of lower-level experts and the horizontal axis representing the anchor points; the elements in the matrix... This represents the activation strength of the i-th low-level expert LoRA when generating the k-th anchor point.

[0112]

[0113] in, At time step t, the i-th low-level expert LoRA is at time step t. Layer activation strength, This indicates the number of tokens contained in the k-th anchor. This represents the set of time steps for generating the k-th anchor point.

[0114] In implementation, color coding can be used to represent activation intensity. For example, a gradient color mapping can be used, such as blue (low intensity) → yellow → red (high intensity), where the color depth of each cell is proportional to the intensity of the color. .

[0115] During implementation, when the user hovers the mouse over a cell, the anchor point corresponding to the cell, the name of the low-level expert LoRA, and its activation strength are displayed. When the cell is clicked, the activation strength distribution curve of that low-level expert LoRA at each layer when the anchor point was generated is displayed on the right or in a pop-up window (the horizontal axis represents the layer). The vertical axis is ).

[0116] The anchor-expert heatmap allows users to clearly see the changes in the model's focus when generating different anchors. This time-dimensional visualization reveals how the model dynamically adjusts the experts relied upon for subsequent reasoning based on the generated cognitive results, reflecting the brain-like "top-down" attention regulation.

[0117] In practice, the network layer-expert heatmap is a two-dimensional matrix. The vertical axis represents the low-level expert (LoRA), and the horizontal axis represents the network layers of the brain-like target recognition model. It is used to observe the changes in expert roles in networks of different depths. The elements in the matrix... This indicates that the i-th low-level expert LoRA is at the . The activation intensity of the layer.

[0118]

[0119] in, At time step t, the i-th low-level expert LoRA is at time step t. Layer activation strength, This indicates the number of tokens contained in the k-th anchor. This represents the set of time steps for generating the k-th anchor point.

[0120] In implementation, color coding can also be used to represent activation intensity. For example, a gradient color mapping can be used, such as blue (low intensity) → yellow → red (high intensity), where the color depth of each cell is proportional to the intensity of the color. .

[0121] In implementation, the network layer-expert heatmap includes interactive functionality. When the user hovers the mouse over a cell, it displays the corresponding network layer number, the name of the low-level expert LoRA, and its activation intensity. Clicking a cell displays the activation intensity distribution curve of that low-level expert LoRA at that layer when generating each anchor point, either on the right or in a pop-up window (the horizontal axis represents the anchor point). The vertical axis is ).

[0122] In practice, the network layer-expert heatmap can also display the activation intensity distribution of each low-level expert LoRA in different network layers at a selected anchor point or time step, without performing cross-anchor point averaging.

[0123] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0124] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A visualization method for a brain-like target recognition reasoning chain, characterized in that, Includes the following steps: A trained brain-like target recognition model is obtained; the brain-like target recognition model includes multiple low-level expert LoRAs and a total expert LoRA; each low-level expert LoRA corresponds to a low-level visual cognitive task, and the total expert LoRA is used to activate the low-level expert LoRAs and perform feature interaction; Input the image to be recognized and the instructions into the brain-like target recognition model; During the reasoning process of the brain-like target recognition model on the input image, the intermediate state data generated inside the brain-like target recognition model is captured; The intermediate state data includes: cognitive anchor sequence, activation weights of each low-level expert LoRA on different network layers at each time step, and interaction weights between expert LoRAs; A visualization is generated based on the intermediate state data, and the visualization includes a directed graph of expert interaction and / or a spatiotemporal heatmap of expert activation intensity.

2. The visualization method for brain-like target recognition reasoning chain according to claim 1, characterized in that, The expert interaction directed graph is used to show the direction of information flow and interaction intensity among all active experts when generating each anchor point; The nodes in the directed graph of expert interaction represent active expert LoRAs, the size or color of the node represents the interaction influence of the expert LoRA, the directed edges represent the information interaction between active expert LoRAs, and the width or color of the edge represents the total interaction weight between expert LoRAs.

3. The visualization method for brain-like target recognition reasoning chain according to claim 2, characterized in that, The total interaction weights among experts using LoRA are obtained in the following way: Where L represents the number of layers in the brain-like target recognition model, This indicates that at time step t, the distance from expert i's LoRA to expert j's LoRA is at time step t. Layer interaction weights Represents the t-th time step. A set of expert LoRA sets for layer activation. This indicates the number of tokens contained in the k-th anchor. This represents the set of time steps for generating the k-th anchor point. Indicates an indicator function, Indicates a relationship.

4. The visualization method for brain-like target recognition reasoning chain according to claim 1, characterized in that, The spatiotemporal heatmap of expert activation intensity is used to show the activation intensity of each expert LoRA at different time steps and / or different network layers. The spatiotemporal heatmap of expert activation intensity includes the anchor-expert heatmap and the network layer-expert heatmap; The anchor-expert heatmap is a two-dimensional matrix diagram, with the vertical axis representing the LoRA (Lower Expert Registry) and the horizontal axis representing the anchor points; the elements in the matrix... This represents the activation strength of the i-th low-level expert LoRA when generating the k-th anchor point; The network layer-expert heatmap is a two-dimensional matrix, with the vertical axis representing the low-level expert LoRA and the horizontal axis representing the network layers of the brain-like target recognition model. The elements in the matrix... This indicates that the i-th low-level expert LoRA is at the . The activation intensity of the layer.

5. The visualization method for brain-like target recognition reasoning chain according to claim 1, characterized in that, The trained brain-like target recognition model was obtained using the following method: Multiple low-level expert LoRAs are introduced on the pre-trained base model. Each low-level expert LoRA corresponds to a low-level visual cognition task. The corresponding low-level expert LoRA is trained based on the training dataset corresponding to each low-level visual cognition task. Introduce a total expert LoRA on the pre-trained base model; The overall expert LoRA is used to extract latent features, and low-level expert LoRAs are dynamically activated based on the latent features. Enhanced features are generated through feature interaction based on the activated low-level expert LoRAs. Constructing a supervised training dataset; Train the LoRA of the total expert on the process-supervised training dataset; A well-trained brain-like target recognition model is obtained.

6. The visualization method for brain-like target recognition reasoning chain according to claim 5, characterized in that, Based on the dynamic activation of low-level expert LoRA using latent features, and the generation of enhanced features through feature interaction based on the activated low-level expert LoRA, the following steps are performed: For each layer of the pre-trained base model: Whether to activate a low-level expert LoRA in a layer is determined based on the similarity between the latent features of the total expert LoRA and each low-level expert LoRA in that layer. Enhanced features for this layer are generated by feature interaction between the latent features of the activated low-level expert LoRA and the total expert LoRA.

7. The visualization method for brain-like target recognition reasoning chain according to claim 6, characterized in that, Enhanced features are generated through feature interactions between the latent features of the activated low-level expert LoRA and the total expert LoRA, including: Based on the activated expert LoRA, the latent features of the activated expert LoRA are interacted with to obtain interactive features through the up-dimensional matrix of this layer; The interaction weights of the interaction features are obtained by dynamically assigning weights to the interaction features. The enhanced features of this layer are obtained by adding the output features of the pre-trained base model at this layer to the interaction features weighted by interaction weights.

8. The visualization method for brain-like target recognition reasoning chain according to claim 7, characterized in that, Based on the activated expert LoRA, the latent features of the activated expert LoRA are interacted with in this layer's increased-dimensional matrix to obtain interactive features, including: Each activated expert LoRA is designated as the source expert, and each other activated expert LoRA (excluding the source expert itself) is designated as the target expert. The latent features of the source experts are transformed using a trainable interaction matrix; The transformed features are projected through the target expert's upscaling matrix to obtain the interaction features corresponding to the source expert and the target expert.

9. The visualization method for brain-like target recognition reasoning chain according to claim 8, characterized in that, The interaction weights of the interaction features are obtained by dynamically assigning weights to the interaction features, including: The latent features of the overall expert LoRA are used as the query vector; For each interaction feature, calculate the key vector using the up-dimensional matrix of the source expert corresponding to that interaction feature; The interaction weight of this interaction feature is calculated based on the query vector and key vector.

10. The visualization method for brain-like target recognition reasoning chain according to claim 5, characterized in that, Each low-level expert LoRA consists of multiple sets of trainable low-rank matrix pairs, each set of trainable low-rank matrix pairs is set in each layer of the pre-trained base model, and is used to perform low-rank updates on the original weights in each layer of the base model.

Images