Active operator subspace reasoning method based on sparse transcoder
The active operator latent space reasoning method using sparse transcoders solves the problems of high computational overhead and low semantic transparency in the reasoning process of large language models, and realizes an efficient, interpretable and causally controllable reasoning process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies suffer from high computational overhead, high latency, and lack of semantic transparency and causal control during the reasoning process of large language models. Dense latent space reasoning methods cannot participate in the dynamic reasoning process, and sparse feature learning cannot provide proactive semantic intervention.
An active operator latent space reasoning method based on sparse transcoders is adopted. Through latent space trajectory mapping, bidirectional decomposition transition transcoders and composite loss function optimization, the ordered evolution of sparse semantic features is realized, and the sparse features are used to explicitly control the reasoning process.
It achieves high computational and storage efficiency in the reasoning process, improves the mechanical interpretability and causal control of the model, enables precise intervention through sparse features, shortens the reasoning length and reduces the computational dimension.
Smart Images
Figure FSA0000302671220000011 
Figure FSA0000302671220000012 
Figure FSA0000302671220000013
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method for achieving efficient and interpretable reasoning of large language models by elevating a sparse transcoder into an active reasoning operator. Background Technology
[0002] Traditional CoT reasoning relies on explicit lexical generation, which introduces significant computational overhead and inference latency in multi-step reasoning. While existing latent space reasoning methods (such as Coconut or CoLaR) reduce costs by compressing the reasoning process into continuous latent states, their reliance on "dense representations" renders the reasoning process "black box," lacking semantic transparency and causal control. Furthermore, existing sparse feature learning (such as SAE) primarily serves as a posteriori diagnostic tool and cannot participate in the model's dynamic reasoning process. Therefore, there is an urgent need in this field for a reasoning framework that maintains latent space compression efficiency while providing proactive semantic intervention capabilities. Summary of the Invention
[0003] The purpose of this invention is to provide an active operator latent space reasoning method (LSTR) based on a sparse transcoder. By constructing a bidirectional decomposition latent space transition transcoder (LTT), the reasoning process is transformed into the ordered evolution of sparse semantic features. The main technical solution includes the following steps:
[0004] Step S1, Latent Space Trajectory Mapping and Alignment: Map the explicit inference word sequence to the target latent space trajectory Z according to a preset compression ratio r. * Square root pooling is used to aggregate the embedded vector blocks, as shown in the formula: This step aims to eliminate modulus expansion under different compression rates, ensuring the numerical stability of latent space targets.
[0005] Step S2, Latent Space Transition Reasoning Based on Bidirectional Decomposition: State transitions are performed using a Latent Space Transition Transcoder (LTT). The LTT decomposes each latent space step into: a Skip Path: modeling the smooth continuity of the latent space manifold and capturing background drift; and a Sparse Path: injecting local semantic updates using a Top-k sparse activation mechanism. The prediction formula is:
[0006] Step S3, Composite Target Optimization and Feature Resurrection: Training is performed using a composite loss function: L latent =L FVU +λ s L skip +λ g L ghostAmong them, L FVU (Unexplained variance proportions) provide scale-invariant supervisory signals; L ghost Ghost gradients are used to revive inactive neurons under hard sparse constraints, improving dictionary utilization.
[0007] Step S4, Semantic Resolution Control in the Inference Phase: During the inference phase, the number of active semantic features is dynamically managed by adjusting the sparse budget k. The value of k directly determines the granularity of inference and the allocation of computational resources, thus achieving explicit adjustment of inference capacity.
[0008] The present invention has the following advantages:
[0009] (1) Computation and storage efficiency: Compared with CoT, the inference length is shortened by an order of magnitude; compared with dense latent space inference, sparse operators reduce the effective computation dimension.
[0010] (2) Mechanical interpretability: latent space transformations are driven by discrete sparse features with explicit semantics (such as arithmetic operators and logical nodes).
[0011] (3) Causal control: Allows precise rewriting of the model's inference trajectory by intervening in the activation values of specific sparse features. Attached Figure Description
[0012] Figure 1 This is a schematic diagram comparing the explicit thought chain reasoning (CoT), dense latent space reasoning, and the LSTR framework of this invention in terms of reasoning path, computational overhead, and interpretability in embodiments of this invention; the diagram illustrates the principle of LSTR achieving efficient and semantically controllable reasoning through sparse latent space features.
[0013] Figure 2 This is an overall flowchart of the active operator latent space inference method based on sparse transcoders provided in this embodiment of the invention. The diagram shows in detail the compression process from explicit lexical sequences to latent space trajectories, the linear transmission and sparse innovative bidirectional decomposition architecture of the latent space transition transcoder (LTT), and the iterative latent space inference flow.
[0014] Figure 3 This is a schematic diagram of the mechanism analysis and causal intervention of sparse latent space reasoning in an embodiment of the present invention; the left side of the figure shows the sparse feature activation evolution under the correct reasoning trajectory, and the right side shows the experimental process of causal rewriting of the logical path by artificially enhancing specific sparse features. Detailed Implementation
[0015] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of the invention. Figure 2 It can be seen that the method includes the following steps:
[0016] Step S1: Latent space trajectory mapping and alignment. This step establishes the mapping relationship between the explicit inference chain and the compressed latent space target. Specifically, the system receives the explicit inference lexical sequence T. r It is mapped to the target latent space trajectory according to the preset compression ratio r. For each block containing a set β of r word embedding vectors t Aggregation is performed using square root pooling: This aggregation method effectively maintains the variance scale and prevents latent state modulus expansion caused by increasing compression ratio. Subsequently, the mapped latent space target is standardized using the empirical embedding variance pre-computed from the training corpus to ensure the numerical stability of subsequent prediction tasks.
[0017] Step S2: Construction of the Latent Space Transition Transcoder (LTT). This step constructs the core inference operator of this invention for predicting subsequent latent states. This embodiment introduces trainable parameters based on a large language model with frozen parameters (such as Llama-3.2-1B-Instruct) using Low-Rank Adaptation (LoRA) technology. The LTT decomposes the latent space transition process into two functionally decoupled paths: the Linear Transport Path, which employs a bias-independent linear adapter W... skip Modeling smooth drift of latent space manifolds; Sparse Innovation Path: Projecting activation values onto a high-dimensional dictionary space (dimension d) feat =α·d model It implements hard sparsity constraints through a Top-k selection operator to capture fine-grained inference primitives and predict the next latent state. The calculation formula is: Among them, s t It is a sparse feature vector whose activation follows the Top-k mechanism and ReLU nonlinear transformation.
[0018] Step S3: Composite objective optimization and random sampling supervision. This step ensures the stability and semantic consistency of latent space inference through multi-task learning. This embodiment uses a composite latent space loss function for optimization: L latent =L FVU +λ s L skip +λ g L ghost In practice, the variance proportional loss L was not explained. FVU Provides a scale-invariant prediction error metric; skip path alignment loss Lskip Forced linear path absorbs dominant manifold drift; ghost gradient loss L ghost The system utilizes reconstructed residuals to guide the revival of inactive coding units, thereby improving dictionary utilization. Simultaneously, at each latent space step t, the system randomly predicts a word within the block, anchoring the continuous latent space dynamics to discrete linguistic semantics through a random sampling strategy, thus preventing deviations in the inference trajectory.
[0019] Step S4: Semantic Resolution Control during Inference. This step enables dynamic management of the model's inference capacity. In this embodiment, the system supports dynamically changing the number of active semantic features by adjusting the sparse budget k during inference without retraining the model. Experiments show that as k increases, the model's inference accuracy steadily improves without significantly changing the average number of inference steps. This mechanism gives the LSTR system the ability to adjust the trade-off between inference granularity and computational cost in real time.
[0020] Step S5: Causal Verification of Defense Effectiveness and Mechanism. To verify the effectiveness of the technical solution of this invention, mainstream mathematical reasoning benchmark sets (such as GSM8k-Aug, MATH) are used for evaluation. In the embodiment, the system verifies the effectiveness of sparse features through causal feature intervention experiments: by artificially enhancing the activation values of sparse features related to arithmetic operators in specific early steps, a causal shift in the model's reasoning trajectory is observed, ultimately leading to a change in the output result, proving that sparse features, as active operators, dominate the logic generation process. In addition, the system exhibits a clear "pre-causal necessity" characteristic, that is, the core semantic decisions are mainly concentrated in the first 30% of the reasoning process.
Claims
1. A latent space reasoning method for large language models based on sparse transcoders, characterized in that, Includes the following steps: Step S1, Latent Space Trajectory Mapping and Alignment: Receive the explicit inference lexical sequence T r Map it to a compressed latent space trajectory target according to a preset compression ratio r. The process involves several steps: Step S2, Bidirectional latent space transition: A multi-step latent space transition is performed using a latent space transition transcoder (LTT), decomposing each latent space inference step into a linear transmission path and a sparse semantic innovation path. The linear transmission path is used to maintain the continuity of the latent space manifold, while the sparse semantic innovation path is used to inject local semantic updates. Step S3, Semantic resolution adjustment and generation: During the inference process, the number of active semantic features is dynamically controlled by adjusting the sparse budget k, and the output response is predicted at the end point or intermediate step of the latent space trajectory using the language model head.
2. The method according to claim 1, characterized in that, The specific method for mapping the latent space trajectory in step S1 is as follows: For each block containing a set β of r embedding vectors... t Aggregation is performed using square root pooling, with the following formula: Where e j This is the embedding vector of the inference lexical.
3. The method according to claim 1, characterized in that, The latent space transition transcoder (LTT) is built on a dual-path architecture and predicts the next latent space state. The calculation formulas include: s t =Top-k(σ(W) enc (h t -μ))+b dec ), where W skip For bias-independent linear adapter weights, s t h is the sparse feature vector generated by Top-k selection. t This represents the hidden layer state of the large model.
4. The method according to claim 1, characterized in that, The training process of the method employs a composite latent space loss function L. latent =L FVU +λ s L skip +λ g L ghost , where L FVU For the unexplained variance proportion loss, L skip For the jump path alignment loss, L ghost This is the ghost gradient loss used to maximize dictionary utilization.
5. The method according to claim 4, characterized in that, The unexplained variance proportional loss L FVU Using the scale invariance criterion, the formula is: Used to eliminate numerical magnitude shifts at different model sizes and latent space depths.
6. The method according to claim 1, characterized in that, The training process also includes a random sampling lexical supervision strategy: at each latent space step t, the language model head is used to sample the corresponding lexical block β. t+1 By randomly sampling a word for prediction, the continuous latent space dynamics are anchored to discrete language semantics.
7. The method according to claim 1, characterized in that, The sparse semantic innovation path is achieved by projecting activation values onto a high-dimensional dictionary with dimension d. feat =α·d model , where α is the expansion factor; and the decoder weight column vector is constrained to the unit norm.
8. A large language model latent space reasoning system for performing the method as described in any one of claims 1 to 7, characterized in that, include: The trajectory mapping module converts explicit inference text into a compressed latent space vector sequence; the latent space transition transcoding module has a built-in dual-path LTT operator for performing linear manifold transfer and sparse semantic updates; the capacity adjustment module dynamically adjusts the sparse budget k according to the complexity of the inference task; and the response output module generates the final text answer based on the latent space inference trajectory.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.