Multimodal large model space reasoning data processing method based on geometric manifold space

Abstract

The application discloses a multimodal large model space reasoning data processing method based on geometric manifold adaptation, and belongs to the field of large visual language models. The method comprises the following steps: constructing a geometric manifold space of visual features; performing manifold search to generate a geometric counterfactual reference sample that maximizes semantic conflict; inputting original samples and reference samples into a decoder, calculating geometric-oriented features and superimposing the features to intermediate layer hidden states to realize implicit trajectory correction; dynamically weighting and calibrating original output distribution based on reference output distribution to generate a space reasoning response. Through the double-layer correction mechanism of implicit features and explicit distribution, the application effectively eliminates spatial hallucination without fine-tuning model parameters, and significantly improves the spatial reasoning accuracy of the multimodal large model in complex scenes.

Description

Technical Field

[0001] This application belongs to the field of large-scale visual language models, specifically involving a multimodal large-scale model space inference data processing method based on geometric manifold space. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, large-scale visual language models (... As a multimodal artificial intelligence technology capable of simultaneously processing and understanding image and text information, it has achieved significant breakthroughs in areas such as general visual recognition, human-computer interaction, and content generation. However, despite these models demonstrating excellent general capabilities in describing image content or answering general questions, they still face serious challenges when dealing with reasoning tasks involving fine spatial relationships. Specifically, when faced with instructions asking about the relative positions (e.g., left, right, front, back) or spatial layout of objects, existing models often produce directional illusions that contradict visual facts. This lack of spatial perception severely limits their practical deployment in safety-critical applications such as autonomous driving, embodied intelligent robots, and high-precision navigation.

[0003] To enhance the spatial reasoning capabilities of models, existing technologies primarily employ two approaches. The first approach focuses on enhancements during the model training phase, such as supervised fine-tuning using large-scale 3D spatial data or introducing reinforcement learning strategies based on thought chains. However, these methods are typically data-intensive and computationally intensive tasks. They not only require collecting and labeling massive amounts of spatial location data, but also consume expensive computational resources for retraining or fine-tuning large model parameters. Furthermore, they face the risk of catastrophic forgetting, which could impair the model's original general language capabilities while improving its spatial abilities.

[0004] The second approach focuses on intervention during the inference stage, aiming to suppress hallucinations by adjusting the inference strategy without updating model parameters. This is currently a hot research topic. The most representative existing technique is visual contrastive decoding, such as Leng et al., 2024, “Mitigating object hallucinations in large vision-language models through visual contrastive decoding.” This method typically constructs a visually impaired distorted sample as a negative reference by adding Gaussian noise to the original input image or randomly masking image patches. The model's predictions are then corrected by comparing the output distribution differences between the original and distorted samples. While this method can alleviate some object hallucination problems, it has significant limitations in handling spatial relationships. First, pixel-level noise perturbation or occlusion is an unstructured disruption. While interfering with image information, it inevitably damages key visual semantic features such as texture, shape, and color of the object itself. This makes it impossible for the model to distinguish between changes in spatial location and changes in the object itself during comparison, thus introducing additional semantic noise. Secondly, existing technologies for constructing negative reference samples typically rely on random sampling. This blind perturbation makes it difficult to accurately generate highly discriminative counterfactual samples for specific spatial geometric relationships (such as rotation and translation), resulting in low intervention efficiency. Furthermore, while general inter-layer contrastive decoding strategies can improve linguistic factuality, they often overlook the specificities of visual spatial semantics and fail to address the root cause of spatial illusions: the "representation-output misalignment" problem, where the model correctly encodes spatial features in intermediate layers during decoding, but is ultimately suppressed by strong linguistic prior probabilities.

[0005] Therefore, how to design a model that can precisely manipulate spatial geometric information during the inference stage, and actively discover and utilize the model's potential geometric perception capabilities while strictly preserving the semantics of the object's visual appearance, in order to efficiently and accurately eliminate the illusion of spatial relationships, is a technical challenge that urgently needs to be solved in the field of large-scale visual language model technology. Summary of the Invention

[0006] Firstly, in view of the shortcomings of the existing technology, the purpose of this application is to provide a multimodal large model spatial reasoning data processing method based on geometric manifold space, which actively discovers and utilizes the potential geometric perception ability of the model while strictly preserving the semantics of the visual appearance of objects, so as to efficiently and accurately eliminate the illusion of spatial relationships.

[0007] The objective of this application can be achieved through the following technical solutions: A multimodal large model space inference data processing method based on geometric manifold space, the method comprising: Obtain the visual feature sequence corresponding to the image, and construct a geometric manifold space mapped to the visual feature sequence according to a preset position encoding function, wherein the geometric manifold space is a feature embedding space that characterizes the geometric topological structure of the position coordinates corresponding to the visual feature sequence; A manifold search operation is performed within the geometric manifold space. Candidate coordinates are generated by applying parameterized geometric transformations to the position coordinates. Target geometric parameters are determined based on the semantic conflict index calculated by the multimodal large model for the candidate coordinates. Geometric counterfactual reference samples are generated based on the target geometric parameters. The original sample data of the inference request to be processed and the geometric counterfactual reference sample are respectively input into the decoder of the multimodal large model, and the original hidden state data and reference hidden state data output by the decoder in the preset intermediate layer are extracted. Calculate the geometric guidance features between the original hidden state data and the reference hidden state data, and superimpose the geometric guidance features onto the original hidden state data to generate the corrected hidden state data; Decoding is performed based on the corrected hidden state data to generate an original output distribution, and a distribution difference metric between the original output distribution and the reference output distribution is calculated. The original output distribution is dynamically weighted and calibrated based on the distribution difference metric, and spatial reasoning response data corresponding to the reasoning request to be processed is generated based on the dynamically weighted and calibrated original output distribution. The reference output distribution is a probability distribution generated based on the geometric counterfactual reference sample.

[0008] Secondly, in view of the shortcomings of the existing technology, the purpose of this application is to provide a multimodal large model spatial reasoning data processing device that, while strictly preserving the semantics of the visual appearance of objects, actively discovers and utilizes the potential geometric perception capabilities of the model to efficiently and accurately eliminate spatial relationship illusions.

[0009] The objective of this application can be achieved through the following technical solutions: A multimodal large-model spatial reasoning data processing device, the device comprising: Memory, configured to store computer-executable instructions; The processor is configured to execute the computer-executable instructions, which, when executed, cause the apparatus to perform the method steps as described in the first aspect.

[0010] Thirdly, in view of the shortcomings of the prior art, the purpose of this application is to provide a computer-readable storage medium that, while strictly preserving the semantics of the visual appearance of objects, actively discovers and utilizes the potential geometric perception capabilities of the model in order to efficiently and accurately eliminate the illusion of spatial relationships.

[0011] The objective of this application can be achieved through the following technical solutions: A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method steps as described in the first aspect.

[0012] The beneficial effects of this application are: This application pioneers a geometric manifold space paradigm for inference time. By performing differential search on the manifold constructed by position encoding, it successfully utilizes geometric counterfactual signals to drive implicit trajectory correction and explicit distribution alignment, and dynamically repairs broken perceptual pathways without updating parameters. This application demonstrates outstanding versatility and robustness across scenarios encompassing different location-encoding manifolds and multimodal tasks. This not only verifies the feasibility of awakening the model's latent geometric capabilities through inference-time intervention, but also paves a cost-effective and interpretable path for constructing physically grounded visual language models. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the overall process architecture and data flow of an embodiment of this application; Figure 2 This is a schematic diagram comparing the accuracy of existing models based on the probe curve with the final output. Figure 3 This is a line graph comparing the linear detection accuracy of embodiments of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] The following terms are explained in this application: From a geometric manifold perspective, this application re-formulates the spatial representation mechanism of LVLMs as operations on a geometric manifold. First, the sensitivity of the self-attention mechanism to geometric transformations is defined, and then the encoding paradigms at different locations are analyzed to support the definition of continuous geometric operators in the feature space.

[0017] Self-attention is used as a geometric function, and in this application, a sequence of visual features is given. , Spatial information is aggregated using a self-attention mechanism. Features are first projected as queries, keys, and values. Position encoding function. geometric coordinates Injected into the feature space: , The attention score is defined by the geometrically related inner product: This application treats position encoding as a geometrically parameterized... A function that controls (e.g., rotation angle). Define a geometric transformation operator. The transformed attention score becomes Due to the permutation invariance of the dot product operation, the model's perception of spatial structure depends entirely on the coordinate system. This means that any tiny perturbation applied to the underlying coordinate manifold... They will all pass through Deterministic propagation to the attention matrix This reshapes the spatial attention distribution of the model.

[0018] Parametric positional manifolds: Existing LVLMs employ two main positional encoding paradigms. Despite their different implementation mechanisms, this application reveals that they essentially construct a latent geometrical manifold, allowing us to define differentiable geometric transformations on it. Learnable absolute positional encoding (APE) is also discussed. Examples include... Architectures that utilize learnable parameter matrices .Although It is discrete in storage, but topologically it is... of Image mesh topological homeomorphism. Therefore, this application will The model is performed as a discretized geometric manifold. Let... coordinates Corresponding position vector: From this perspective, this application can define a resampling operator. For any affine transformation parameters (e.g., rotation matrix), transformed position encoding By applying the original matrix to a continuous manifold Interpolation sampling was performed to obtain: This allows this application to transform a discrete parameter matrix into a continuous, optimizable geometric variable.

[0019] Rotational Position Embedding (RoPE). RoPE uses a multiplicative mechanism to encode positions through rotations in the complex field or high-dimensional subspaces. Its encoding function is defined as: in It is based on coordinates A uniquely determined orthogonal rotation matrix. From a group theory perspective, RoPE's core advantage lies in its construction of a continuous manifold with a Lie group structure.

[0020] Specifically, rotation matrix This constitutes a special orthogonal group. Under this mechanism, the attention score depends on the relative coordinate difference: .

[0021] In modern times (For example In the series, coordinates It is extended to a multidimensional form to accommodate different modalities: 2DRoPE: For images, the feature dimension is decomposed into two subspaces, each utilizing... and Use this to encode the height and width.

[0022] 3DRoPE: Introducing a temporal dimension for video tasks. To construct a spatiotemporal coordinate system .

[0023] RoPE's core advantage lies in its geometric equivariance, which adapts well to input coordinates. Any geometric transformation applied Mathematically, this is equivalent to performing a group action on a Lie group manifold. This allows the present application to perform this action without touching the model weights, simply by moving coordinates on the manifold. This allows for precise control of the geometric relationships in the feature space.

[0024] To elucidate the mechanical origin of spatial illusion, this application delves into the model's internal workings to explore the dynamic changes in geometric information during the decoding process. To this end, this application freezes the parameters of the LVLM, extracts the hidden state of the last token at each layer of the decoder, and sets up two sets of linear probing experiments: (1) In-distribution probing: The Controlled Images dataset is randomly divided into a training set (10%) and a test set (90%). (2) Out-of-distribution probing: Utilizing data from real-world scenes... The dataset was trained and evaluated on controlled images with clean backgrounds. Experimental results are as follows: Figure 2 As shown, based on the comparison between the detection curve and the final output accuracy, this application summarizes the following two key findings.

[0025] Discovery 1: Potential geometric capabilities The probe curves reveal a surprising fact: the intermediate layers of LVLMs actually encode spatial geometric information with precision far exceeding the model's final output. For example... Figure 2 As shown, for both LLaVA-1.5 and Qwen2.5-VL, the in-distribution (ID) detection accuracy (blue line) rapidly climbs to nearly 100% in the intermediate layers and remains high. More importantly, the out-of-distribution (OOD) detection accuracy (red line), representing generalization ability, also reaches a significant peak at specific levels (Layer 14 for LLaVA and Layer 28 for Qwen), far exceeding the gray output baseline. This phenomenon strongly refutes the traditional view that "the model lacks spatial awareness." It proves that a high-confidence intrinsic geometric subspace exists within the geometric manifold constructed by position encoding. The model does not "not understand" spatial relationships; rather, these correct visual signals are latent in the representational manifold of the intermediate layers, waiting to be activated by the correct mechanism.

[0026] Finding 2: Language Prior Inhibition Although the intermediate layers have correctly encoded spatial geometry information, this capability results in a severe "representation-output misalignment" when passed to the output layer. (Observation) Figure 2The OOD detection curve (red line) shows that after reaching its peak, the accuracy drops sharply with increasing layer depth. This indicates that general visual geometric representations are overshadowed by strong linguistic priors in the deep decoding stage. The model tends to discard cross-domain, purely geometric evidence and instead rely on language shortcuts with higher statistical probabilities in the training data. This shift from "visual semantic dominance" to "language probability dominance" directly leads to a misalignment between cognition and expression—the model subconsciously "sees" the correct location but, constrained by language habits, "says" the wrong answer.

[0027] Example 1: like Figure 1 As shown, this application discloses a method for processing multimodal large model space inference data based on geometric manifold space, the method comprising: Obtain the visual feature sequence corresponding to the image, and construct a geometric manifold space mapped to the visual feature sequence according to a preset position encoding function, wherein the geometric manifold space is a feature embedding space that characterizes the geometric topological structure of the position coordinates corresponding to the visual feature sequence; A manifold search operation is performed within the geometric manifold space. Candidate coordinates are generated by applying parameterized geometric transformations to the position coordinates. Target geometric parameters are determined based on the semantic conflict index calculated by the multimodal large model for the candidate coordinates. Geometric counterfactual reference samples are generated based on the target geometric parameters. The original sample data of the inference request to be processed and the geometric counterfactual reference sample are respectively input into the decoder of the multimodal large model, and the original hidden state data and reference hidden state data output by the decoder in the preset intermediate layer are extracted. Calculate the geometric guidance features between the original hidden state data and the reference hidden state data, and superimpose the geometric guidance features onto the original hidden state data to generate the corrected hidden state data; Decoding is performed based on the corrected hidden state data to generate an original output distribution, and a distribution difference metric between the original output distribution and the reference output distribution is calculated. The original output distribution is dynamically weighted and calibrated based on the distribution difference metric, and spatial reasoning response data corresponding to the reasoning request to be processed is generated based on the dynamically weighted and calibrated original output distribution. The reference output distribution is a probability distribution generated based on the geometric counterfactual reference sample.

[0028] Specifically, the method of this application embodiment can be applied to electronic devices such as servers, cloud computing nodes, or high-performance personal terminals. This method, without relying on expensive model parameter fine-tuning, utilizes the geometric characteristics of the positional encoding of the Multimodal Large Model (LVLM) itself, locates counterfactual references through manifold difference search, and then drives a dual geometric correction mechanism, achieving a training-free inference-time geometric manifold space.

[0029] In practical implementation, in response to receiving a processing inference request containing images and text, such as asking to determine the color of the object on the left, the system first extracts the visual feature sequence of the image through a visual encoder (such as CLIP-ViT). Based on this, a geometric manifold space mapped to the visual feature sequence is constructed using a preset positional encoding function. The geometric manifold space refers to the feature embedding space defined by positional encoding (especially Rotated Position Embedding, RoPE). Since RoPE preserves the relative positional relationships in the visual feature sequence through vector rotation operations by mapping position indices to rotations on the complex plane, this feature space possesses a Lie group (e.g., ...). The topological structure of a group. Under this structure, transformations of position coordinates (such as rotation and translation) can be equivalent to smooth movements on the manifold.

[0030] Subsequently, the system performs a manifold search operation within the geometric manifold space. The technical objective of this step is to construct a manifold with the original observations. Geometric counterfactual references that create strong semantic contrast In order not to disrupt visual semantics, this application chooses to perform active exploration on the manifold defined by position encoding.

[0031] Specifically, the system applies parametric geometric transformations to the original position coordinates of the image. In a preferred embodiment, the visual spatial perception of LVLMs relies on position encoding. From a group theory perspective, positional encoding essentially constructs a geometric manifold with a Lie group structure. Therefore, this application defines a parameterized geometric transformation operator. By optimizing parameters (Rotation angle) Smoothly moves on the manifold. Let the original coordinates be... Transformed coordinates The calculation is as follows: in The center of the image is the center. This transformation is essentially a special orthogonal group. The group action in spatial coordinates ensures that the generated samples lie strictly on the geometric manifold. This application will... It is compatible with three mainstream encoding methods, ensuring the equivariance of geometric topology: 1DAPE: will Considered as discrete manifold sampling points, based on Perform bilinear interpolation resampling on it.

[0032] 2DRoPE: Utilizing the rotation properties of the complex field, directly based on... Phase transition of the driving frequency basis on the manifold.

[0033] 3DRoPE: For video data, only the spatial coordinates within each frame are updated. Maintain time index Unchanged. This ensures... The flipping only occurs in the spatial geometry, while strictly preserving the original temporal logic and object appearance features.

[0034] The system iteratively optimizes the above geometric parameters. This generates a series of candidate coordinates. For each candidate coordinate, the system calculates a semantic conflict index, which quantifies the model's "confusion level" or "misjudgment tendency" under the geometric transformation. Specifically, the objective function is defined... This application aims to find a perspective from which the model generates extremely high confidence in erroneous counterfactual spatial relations, i.e., induces the greatest semantic conflict: a semantic conflict index. Defined as the probability value ranked first in the model's predicted probability distribution ( ) and the probability value of ranking second ( The difference between ) is The larger the difference, the more confident the model is in its current incorrect prediction, or the more indecisive it is between the two options, i.e., the more severe the semantic conflict.

[0035] To efficiently determine the target geometric parameters (i.e., the optimal interference angle) that maximize the semantic conflict index. In this embodiment, a zero-order optimization algorithm is preferably used for manifold search. Specifically, an opposite sampling strategy is employed to sample pairs of positive and negative Gaussian noise perturbations in the parameter space, and the gradient update direction is estimated using a dominant normalization technique, thereby guiding the search center. The system gradually approximates the globally optimal geometric parameters. Once the search converges, the system uses these target geometric parameters as the basis for its operation. A transformed positional code is generated and combined with the original visual features to construct a "geometric counterfactual reference sample." This sample is visually and semantically similar to the original... Figure 1 The pixel count remains unchanged, but the spatial geometry is distorted to a state where the model is most prone to errors.

[0036] Specifically, among them Indicates the probability of the candidate word distribution. High-dimensional spatial terms. Maximizing this probability margin essentially involves finding a semantic pole on the manifold. Due to the high-dimensional non-convexity of the manifold, this application employs zero-order optimization to perform gradient estimation in the tangent space of the manifold. In each iteration... This application is from the current perspective. Centered on, sample random cutting vectors , obtain candidate state And update the search center using Advantage Normalization: in To update the step size, This is a numerically stable term. The algorithm effectively drives the search trajectory to escape the local gravitational wells formed by linguistic priors, and quickly converges to the optimal counterfactual state on the manifold. .

[0037] The next step involves performing a two-layer correction using this counterfactual reference. The system inputs the raw sample data of the inference request to be processed and the aforementioned geometric counterfactual reference sample into the decoder of the multimodal large model in parallel or serially. The raw hidden state data is extracted from the model's preset intermediate layers (typically deep network layers sensitive to spatial features). and reference hidden state data At this point, the system calculates the geometric guidance characteristics between the two. Specifically, the system first calculates the difference vector between the two and normalizes it using the L2 norm to obtain the unit direction vector. : That is, in obtaining the optimal reference Subsequently, this application performs implicit trajectory correction, aiming to distill pure geometric signals from latent representations and orthogonalize the inference trajectories that have experienced geometric collapse. The distribution of spatial geometric information in each layer of the decoder exhibits dynamic changes, and at the peak layer... Most significant. Therefore, this application calculates the original sample. With reference The normalized difference of the output states of the intermediate self-attention module is defined as the discriminative geometric gradient. : Subsequently, the system introduces a preset correction strength coefficient. The geometric guidance feature is then superimposed onto the original hidden state to generate the corrected hidden state data. This operation forcibly corrects the model's inference trajectory within the feature space, moving it away from the illusory region caused by spatial location encoding.

[0038] In other words, through differential computation, this application cancels out rotationally invariant non-spatial semantics (such as texture and color), retaining only the manifold tangent direction, which is sensitive to coordinate transformations. The reason for choosing a self-attention module instead of a feedforward network (FFN) is that self-attention is a geometric operator responsible for constructing spatial topology, while FFN focuses more on factual knowledge. Stripping the FFN signal minimizes semantic noise interference. Subsequently, to prevent geometric information from drifting during deep transmission, this application selects a peak layer. Implementation of intervention: in To correct the intensity, this intervention mechanism is equivalent to applying a geometric restorative force, pulling the deviated latent representation back onto the manifold with the correct geometric curvature, forcing the model to maintain its perception of spatial structure during deep decoding.

[0039] Finally, the system continues the decoding operation based on the corrected hidden state data to generate the original output distribution. (Logits). Simultaneously, the system also acquires the reference output distribution generated based on geometric counterfactual reference samples. To further eliminate residual distribution bias, the system calculates a measure of the distribution difference between the two distributions, preferably using the Jensen-Shannon divergence (JSD). Based on this divergence value, the system determines a dynamic weight through a mapping function. A larger divergence indicates a significant difference between the original prediction and the counterfactual prediction, requiring stronger calibration. The final dynamic weighted calibration is achieved through a linear interpolation formula: Through the above steps, based on the final distribution after dynamic weighting calibration The generated spatial reasoning response data (i.e. the final text answer) not only eliminates spatial misleading information at the feature level, but also suppresses interference from language priors at the output level, thereby significantly improving the accuracy and robustness of multimodal large models in handling complex spatial relationship tasks.

[0040] That is, although implicit corrections protect internal features, the language priors at the output can still alter the final generated distribution. Therefore, this application introduces explicit distribution alignment. This application will refer to the sample... Treating it as a counterfactual anchor point, we reshape the prediction by utilizing the difference in distribution between it and the original sample in the Logits space. Let... The output Logits of the original sample and the reference are respectively, and the aligned distribution is shown below. Defined as: To suppress hallucinations while maintaining the structural integrity of the original distribution, this application restricts calibration to the candidate word set only and introduces dynamic weights based on Jensen-Shannon divergence. Its sensitivity to geometric transformations is determined by the model's sensitivity. Decide: when When the value is small, it indicates that the model ignores drastic geometric transformations, meaning that the original sample and the counterfactual reference output distribution are highly similar. This implies that the model is in a state of linguistic illusion, where... Automatically amplify and forcefully amplify the geometric difference signal to correct the deviation; Conversely, the intervention weight should be reduced to avoid unnecessary interference with the correct predictions with high confidence.

[0041] To comprehensively and objectively verify the practical technical effect of the data processing method based on geometric manifold adaptation proposed in this application in improving the spatial perception and logical reasoning capabilities of multimodal large models, this embodiment selects a highly challenging visual spatial reasoning benchmark dataset that is widely recognized in academia and industry for quantitative evaluation and comparative analysis.

[0042] Figure 1 This paper illustrates the overall workflow architecture and data flow of the multimodal large model spatial inference data processing method proposed in this application. The processing flow is configured to, upon receiving input containing visual content and text queries, perform three cascaded processing steps—adaptive spatial negative sample construction (Step 1), spatial guidance intervention (Step 2), and dynamic Logits calibration (Step 3)—to finally output the calibrated prediction results.

[0043] First, in the input phase, the system receives multimodal data including visual and linguistic modalities. As shown in the figure, the visual modality can be a static image, such as a scene showing the relative positions of a book and a cap; or it can be a dynamic video, such as a sequence of consecutive frames showing the spatial relationships between a refrigerator, a washing machine, and a stove. The linguistic modality involves spatial reasoning questions based on the aforementioned visual content, such as "Where is the book next to the cap?" or "If I stand next to the refrigerator facing the washing machine, is the stove to my left, right, or behind me?"

[0044] Subsequently, the data flow proceeds to Step 1: Adaptive Space Negative Sample Construction. In this stage, the system first identifies the position embedding type used by the visual encoder, including 1DAPE (Absolute Position Encoding) for one-dimensional sequences, 2DRoPE (Rotation Position Encoding) for two-dimensional images, and 3DRoPE for three-dimensional videos. Based on the identified position embedding type, the system performs geometric perturbation operations within the corresponding geometric manifold space. Specifically, the system introduces an optimizable geometric parameter (i.e., rotation angle). The parameters are then searched and updated using a zero-order optimization algorithm to find the perturbation state that maximizes semantic conflict. Based on the determined parameters... The system generates perturbation coordinates and uses these coordinates to reconstruct visual features, thereby generating negative samples. As shown on the right side of the figure, this negative sample It maintains visual semantic consistency with the original image, but exhibits distorted or misaligned features in spatial geometry (such as rotational deformation of grid lines).

[0045] Next, the data flow proceeds to step 2: spatial guidance intervention. The system will then process the original, undisturbed positive samples. Compared with the negative samples generated in step 1 The decoder layer is used for parallel input of large multimodal models. Among the many layers of the decoder, the system predetermines a peak layer that is most sensitive to spatial features. (As shown in the orange box in the figure). In this peak layer At the output of the system, the positive hidden states corresponding to the positive samples are extracted respectively. and the negative hidden state corresponding to the negative sample. Subsequently, the system performs a subtraction operation (as shown by the minus sign icon in the figure) to calculate the difference vector between the two, i.e., the steering vector. This steering vector is back-injected (as shown by the dashed arrow in the figure) into the model's inference path to suppress spatial illusions induced by incorrect location encoding at the feature level.

[0046] Finally, the data stream proceeds to step 3: dynamic Logits calibration. The feature stream processed in step 2 is then transmitted to the language modeling head. At the output, the system acquires the bias prediction distributions generated based on positive samples. and language prior distributions generated based on negative samples As shown in the figure, in In the distribution, the probability of the incorrect option "right" (red bar) may be significantly higher than the probability of the correct option "left" (green bar), indicating spatial misjudgment in the model. In the distribution, the system captures prior probabilities generated by the model relying solely on language habits (e.g., a blind preference for "right"). The system performs subtraction again (as shown by the minus sign icon in the figure), calculates the difference between the two distributions, and performs dynamic weighted fusion to generate the final calibrated prediction distribution. In the final In the distribution, the probability of the correct option "left" is significantly increased (as shown by the green upward arrow in the figure), while the probability of the incorrect option "right" is effectively suppressed (as shown by the red downward arrow in the figure).

[0047] Finally, in the output phase, the system is based on the calibrated distribution. It outputs the correct answer, such as "left", thereby correcting the spatial hallucination (such as "right") that the original model may produce, and achieves accurate processing of multimodal spatial reasoning tasks.

[0048] experiment: Experimental setup Benchmarking. To comprehensively verify the data processing method proposed in this application ( To assess the effectiveness of this approach in image-based spatial reasoning tasks, this embodiment evaluates image-based spatial reasoning capabilities on the WhatsUp (Kamath et al., 2023) and VSR (Liuet et al., 2023) benchmarks. WhatsUp comprises a synthetic domain and a ground truth domain: synthetic data is divided into Controlled A (a pair of objects of different sizes) and Controlled B (two small objects); while ground truth data from MS COCO (Lin et al., 2014) and Visual Genome (VG) (Krishna et al., 2017) are categorized into one-object and two-object subsets. Following the AdaptVis (Chen et al., 2025b) setup, we refactor these into a generative multiple-choice question-answering task, applying a four-option setting (left, right, up, down) to the Controlled and COCO subsets, and extending the VG to a six-option setting (left, right, up, down, front, back). For VSR, we used the question-answering format generated by GPT-4o (Hurst et al., 2024) to evaluate the model's generative spatial understanding capabilities. Furthermore, for video-based spatial reasoning, we selected the object localization subtask from VSI-Bench (Yang et al., 2025) to evaluate the model's ability to accurately determine relative object positions in temporal dynamics.

[0049] Evaluation Metrics. For all benchmarks, we report the exact match accuracy for all benchmarks. For VSR, we additionally report the F1 score. Furthermore, leveraging the unique contrastive properties of the controlled images subset in WhatsUp, we introduce consistency metrics to eliminate random guessing: Pair Accuracy requires the model to correctly answer both queries in a mirrored spatial relation pair, while Set Accuracy requires correct predictions for all spatial relations within a specific scene.

[0050] Baseline method. This application will It was compared with existing representative inference-time intervention methods, including contrastive decoding strategies such as DoLa and VCD, and attention-based intervention methods such as AdaptVis. All baseline methods were set up according to the default configurations in their original papers.

[0051] Implementation Details. This application validates the effectiveness of its method on three mainstream LVLMs, covering different positional encoding manifolds: LLaVA-1.5 7B (using 1D APE), Qwen2.5-VL3B, and Qwen3-VL 8B (for video tasks, using 3D RoPE), with greedy decoding as the default inference strategy.

[0052] Main results: Table 1. Benchmark tests of the GRASP method in image spatial inference Table 1. Key results of WhatsUp benchmark tests on the Controlled, COCO, and VG subsets. The best-performing method for each model and subset is highlighted in bold. PAcc and SAcc represent PairAccuracy and SetAccuracy, respectively.

[0053] Image benchmark results. Table 1 shows the comparison results on the WhatsUp benchmark. It consistently outperforms baseline methods across various domains. Notably, in... ( )superior, exist Implemented on subsets The peak accuracy was improved, and the set accuracy was increased from [previous value]. Increased to This significant improvement in set-level consistency validates the effectiveness of implicit trajectory correction in locking onto intrinsic geometric truth to combat unstable linguistic priors. compared to, It exhibits superior robustness in real-world scenarios. The above has been implemented The gain. Furthermore. Its effectiveness under different positional coding paradigms has been demonstrated: in stronger... trunk( On top of that, it further... Subset performance improved ,achieve The accuracy of this study confirms the universality of manifold differential search on both discrete and continuous manifolds. In contrast, general illusion mitigation methods such as... and They failed to provide meaningful improvements because they primarily addressed linguistic uncertainty while ignoring the underlying cause of geometric misalignment.

[0054] To verify the versatility of the proposed manifold differential search algorithm in processing temporal dynamic data, this embodiment further evaluates it on the object localization subtask of the video spatial reasoning benchmark VSI-Bench. The experiment uses the Qwen3-VL 8B, which supports 3D Rotation Position Encoding (3D RoPE), as the base model to test the method's adaptability to high-dimensional continuous manifolds.

[0055] Table 2. Generalization ability of the GRASP method on video benchmarks As shown in Table 2, the experimental results demonstrate that the GRASP method achieves significant performance gains in video relative orientation determination tasks of varying difficulty. In Easy mode, the model's accuracy improved from 48.4% to 56.2%, an increase of 7.8%. In Medium mode, accuracy improved from 49.2% to 52.4%, an increase of 3.2%. Crucially, the method demonstrated exceptional robustness in Hard mode (which typically involves complex camera movements or object occlusion), significantly improving accuracy from 35.9% to 45.6%, an absolute improvement of 9.7%.

[0056] This result strongly demonstrates that the geometrically adapted data processing method (GRASP) proposed in this application can not only calibrate static position encoding in 2D image space, but also effectively adapt to 3D video manifolds that include a temporal dimension. By performing a zero-order optimization search in the manifold space defined by 3DRoPE, this method successfully captures and corrects deep spatial semantic conflicts caused by temporal dynamic changes, thereby significantly enhancing the spatiotemporal reasoning capability of multimodal large models in dynamic scenes.

[0057] To verify the effectiveness and synergistic effect of the proposed dual-layer correction mechanism (i.e., implicit trajectory correction (ITC) and explicit distribution alignment (EDA)), detailed ablation experiments were conducted in this embodiment (as shown in Table 3): ablation experiment Component effectiveness. To verify the contribution of the two-layer correction in this application, this application... Ablation studies were conducted on subsets (Table 3): (1) Implicit trajectory correction (ITC) awakened latent capabilities. For ,only This brought Accuracy gain (from) arrive This confirms the existence of a suppressed potential geometric subspace; These dormant signals were effectively extracted, and the inference trajectory was corrected back to the correct manifold. (2) Explicit Distribution Alignment (EDA) bridged the representation-output gap. In a stronger On the main trunk, Superior ( vs. This indicates that for models with robust internal representations, the main bottleneck shifts to the representation-output misalignment. This barrier is effectively broken by punishing the illusion of language dominance at the output stage. (3) Synergistic effect of GRASP. The complete framework achieves optimal spatial consistency, Set accuracy improved to (Comparison only uses) time This indicates that the awakened geometric signal survives during the decoding process and requires synchronous intervention: both to correct the trajectory path and to align the output exit.

[0058] Table 3: Intervention intensity ( The impact of (1) Implicit Trajectory Correction (ITC) awakens latent capabilities. For ,only This brought Accuracy gain (from) arrive This confirms the existence of a suppressed potential geometric subspace; These dormant signals were effectively extracted, and the inference trajectory was corrected back to the correct manifold.

[0059] (2) Explicit Distribution Alignment (EDA) bridges the representation-output gap. In a stronger... On the main trunk, Superior ( vs. This indicates that for models with robust internal representations, the main bottleneck shifts to the representation-output misalignment. This barrier is effectively broken down by punishing the illusion of language dominance during the output phase.

[0060] (3) The synergistic effect of GRASP. The complete framework achieves optimal spatial consistency, Set accuracy improved to (Comparison only uses) time This indicates that the awakened geometric signal survives during the decoding process and requires synchronous intervention: both to correct the trajectory path and to align the output exit.

[0061] To further explore how the data processing method (GRASP) proposed in this application corrects spatial illusions within the model, this embodiment conducts in-depth visualization verification from two dimensions: feature representation and language output, through layer-wise probing and probability distribution analysis.

[0062] 1. Feature recovery analysis of the intervention target layer To verify whether the implicit trajectory correction (ITC) in this application successfully recovers the correct geometric signals within the model, this embodiment designs an experiment called "Recovery of Intrinsic Geometric Signals". We trained a set of linear probes to detect whether the hidden states of each layer of the model contain the correct spatial location information.

[0063] like Figure 3 The linear detection accuracy comparison line graph shown has the horizontal axis representing the layer depth of the Transformer decoder model and the vertical axis representing the accuracy of the detector in determining spatial location.

[0064] Baseline performance (dashed line): During the initial inference process, as the number of layers increases, especially after passing through the middle layers, the linear separability of the model for spatial locations shows a decreasing trend (as shown by the gray dashed line in the figure). This indicates that the model gradually loses the geometric information in the visual features and instead relies on linguistic priors.

[0065] Intervention performance (solid line): After using the method of this application to overlay geometrically guided features on a specific intervention target layer (such as the layer marked in yellow in the figure), the detection accuracy showed a significant leap (as shown by the green solid line in the figure).

[0066] For example, in the LLaVA-1.5 model, the peak accuracy of the intermediate layer after intervention once exceeded 80%, far higher than the baseline of about 60%, demonstrating the effectiveness of instantly "awakening" spatial features in the intermediate layer. For the Qwen2.5-VL model: (e.g.) Figure 3As shown in the lower part, the baseline detection accuracy of this model (gray dashed line) experiences a significant performance collapse in the final stages of the network (e.g., around layers 30-35), with accuracy dropping to approximately 55%. In contrast, the controlled detection accuracy (green solid line) using the method described in this application demonstrates extremely strong robustness in deep networks, maintaining a high accuracy of over 80% even at the output labeled "Recovery." This comparison strongly demonstrates that implicit trajectory correction not only works in intermediate layers but also effectively prevents the catastrophic loss of geometric information during deep decoding.

[0067] Conclusion: The line graph visually demonstrates that by introducing geometric counterfactual reference samples and extracting geometric guidance vectors, this application successfully "awakens" and recovers suppressed intrinsic geometric signals deep within the model, enabling the model to reuse these key visual-spatial cues in subsequent inference calculations.

[0068] In some publications, regarding GRASP: Geometric Correction for Active Spatial Awareness A. Mathematical Derivation and Formal Proof This application provides a detailed mathematical foundation supporting the geometric manifold space paradigm for inference presented in the main text. It first formalizes the topological structure of positional manifolds constructed using APE and RoPE, then derives the exact formulas for geometric operators in GRASP, and finally provides a theoretical proof of the manifold differential search algorithm.

[0069] A.1. Topological Analysis of Positional Manifolds This application assumes that positional encoding constructs an embedding in the feature space. Geometric manifolds in Here, this application formally analyzes this manifold structure of Absolute Position Encoding (APE) and Rotational Position Embedding (RoPE).

[0070] A.1.1. Rotational Position Embedding (RoPE) as a Lie Group Manifold Proposition 1. The operations of RoPE define a continuous geometric manifold that is isomorphic to Lie group structure of group direct product.

[0071] Proof. Recall that for position indexes... and eigenvectors RoPE applies a rotation matrix .for For the standard implementation of even numbers, the feature space is divided into Subspaces. Rotation matrix. It is a block diagonal matrix: Each block Corresponding to the frequency band And defined as: The matrix It is a special orthogonal group An element of represents all rotations in 2D Euclidean space.

[0072] Since the block diagonal matrix is ​​composed of The entire transformation is composed of components. Residing in product group ( In the (times). A key property of Lie groups is geometric isotropy. Let geometric transformation coordinates Translation The transformed embedding is: Due to the homomorphic property of exponential mappings in Lie groups, .therefore: This proves the translation on the coordinate manifold (will) Become In mathematics, this is equivalent to applying group actions (rotations) on a characteristic manifold. This is about location. The continuous differentiability of the gradient allows this application to define a continuous gradient on the manifold, thus proving that the manifold differential search of this application is reasonable.

[0073] A.1.2. Absolute Position Encoding (APE) as a Discretized Manifold Definition 1 (Discretized manifold). This application uses the APE parameter matrix... Defined as a continuous function Discrete sampling, where It is a continuous coordinate space.

[0074] In standard visual Transformers (e.g., ViT, LLaVA), It is learnable. However, in order to perform geometric transformations (such as rotations), integer coordinates... Mapping to non-integer coordinates This application must define a continuous extension of the manifold.

[0075] This application employs bilinear manifold interpolation. Let These are the transformed continuous coordinates. Embedded vector. By The computation is performed by projecting onto a grid formed by the four nearest discrete neighbors: top left Top right Bottom left and the lower right .

[0076] GRASP: Geometric Correction for Active Spatial Awareness Embedding on a continuous manifold is defined as: Among them, weight It is the area relative to the rectangle, for example, This interpolation ensures the function Everywhere It is continuous and almost everywhere differentiable (except for integer grid lines), allowing gradient estimation via zero-order optimization.

[0077] A.2. Derivation of Geometric Operators exist In this application, operators are defined. To traverse the manifold. Here, this application derives the exact transformation matrix for 2D spatial data (images) and 3D spatiotemporal data (videos).

[0078] A.2.1. 2D Spatial Operators (Images) For height is Width is The image, in this application, defines the center of rotation as... Let the original coordinates be... Rotation operator surround the coordinate system Rotation angle Transformed coordinates Derivation using standard affine rotation matrices: Boundary handling: due to the rotated coordinates It may fall into the original image domain. In addition, this application applies a reflection-filling strategy on the manifold. Mathematically, for the constraint of... The dimension, if the coordinates This application maps it to (if )or (if This ensures that queries always remain within the defined embedded lookup table.

[0079] A.2.2. 3D Spacetime Operators (Video) For video input (e.g., in) or (In the middle), the coordinate system is 3-dimensional: ,in It is a time frame index.

[0080] A key design choice is the time-preserving constraint. Since the goal of this application is to correct spatial illusions (e.g., "left / right") without breaking the causal logic of the video (e.g., "action A occurs before action B"), the geometric operators must act strictly on the spatial subspace.

[0081] This application will use 3D operators Defined as acting on the augmented coordinate vector The block matrix: in and Handle the translation back to the center. This ensures... ,and It undergoes spatial rotation. Therefore, for 3DRoPE, the temporal frequency components remain unchanged, preserving the temporal coherence of the video.

[0082] A.3. Theoretical Proof of Manifold Differential Search The core involves using zero-order optimization (ZOO) to maximize the semantic conflict target. This application provides a theoretical basis for why such gradient estimation is efficient on geometric manifolds.

[0083] A.3.1. Gradient estimation using Gaussian smoothing Due to the objective function Involving The complete forward propagation and discrete token selection are performed, and the token is either non-differentiable or has broken gradients. This application uses a Gaussian smoothing function. To approximate : GRASP: Geometric Correction for Active Spatial Awareness in This is the smoothing parameter. According to Nesterov's theorem regarding stochastic gradient-free minimization, the gradient of this smoothing function is well-defined and can be calculated as: A.3.2. Derivation of the Update Rule This application utilizes a variance reduction estimator. Instead of sampling a single sample, this application samples within the tangent space of the manifold. one direction . The update rule in this application is derived from the opposite sampling principle used to reduce variance. However, given the low dimensionality of the search space in this application (rotation angle of 1D), this application simplifies this to a population-based estimation.

[0084] The update term in formula (6) uses dominance normalization: Substituting this into the gradient estimator, the update step becomes: This served as The natural gradient descent approximation in the distributed parameter space effectively readjusts the step size according to the local curvature of the semantic landscape. This ensures that the search converges efficiently to a point where... The "geometric counterfactual reference" for marginal maximization.

[0085] In some publications, this application can be discussed through the following process: 1: Input: Visual input (Image or video), visual encoder Scoring function 2: Hyperparameter: Learning rate Noise scale Group size Maximum number of steps Attenuation factor Stability term 3: Initialization: Search Center Optimal angle Maximum score 4:for to do 5: {Step 1: Group Sampling (Opposite Sampling)} 6: sampling Original noise item 7: Constructing symmetric sets: 8: Calculate scaling perturbation: for 9: Generate candidate angles: 10: {Step 2: Group Assessment} 11: for to do 12: use Constructing negative samples 13: Obtain the probability distribution: 14: Calculate the score: 15: if then 16: {Tracing the Global Optimum} 17: end if 18: end for 19: {Step 3: Advantage Normalization} 20: Calculate the mean: twenty one: Calculate the standard deviation: twenty two: Computational advantages: twenty three: {Step 4: Weighted Update} twenty four: Estimated update direction: 25: Update Center: 26: Decrease learning rate: 27:end for 28: Return: Best Angle and negative samples Example 2: This application further provides a multimodal large model spatial inference data processing device. At the physical level, this device is an electronic device integrating high-performance computing capabilities. Its core hardware architecture includes at least one processor and a memory communicatively connected to the processor. The memory is configured to store computer-executable instructions, and the processor, by executing these instructions, can transform the logical algorithms in the aforementioned method embodiments into specific physical electrical signal operations, thereby achieving geometric manifold adaptation and calibration of the multimodal large model inference process.

[0086] In the specific hardware implementation, the processor is not limited to a single central processing unit (CPU), but preferably adopts a heterogeneous computing architecture to adapt to the needs of multimodal large model (LVLM) for parallel computation of massive parameters. Specifically, the processor may include a general-purpose processor for performing logic control and task scheduling, and an AI acceleration unit, such as a graphics processing unit (GPU), tensor processing unit (TPU), or neural network processor (NPU), specifically for performing high-dimensional tensor operations and differential search. When the device is running, the processor first responds to the received inference request containing image and text instructions through the input / output interface, loading image data into the video memory. Subsequently, the processor calls preset visual encoder instructions to extract features from the image and generate a high-dimensional visual feature sequence. In this process, the processor uses the matrix operation logic corresponding to the position encoding function (such as Rotation Position Embedding, RoPE) to construct a geometric manifold structure in the feature embedding space. This manifold space is represented in physical storage as a set of floating-point matrices with specific topological properties, characterized by the ability to change position coordinates while maintaining semantic coherence through parameterized geometric transformations (such as rotation matrix multiplication).

[0087] To eliminate spatial illusions, the processor is configured to perform zero-order angle optimization operations within the constructed geometric manifold space. This process physically manifests as frequent calls to the random number generator and gradient estimation logic. Specifically, the processor executes contrastive sampling instructions to generate raw noise following a standard normal distribution and its negative symmetric term, and calculates a decay coefficient based on the current clock cycle or iteration step, thereby generating paired scaling perturbation data. The processor superimposes this perturbation data into the search center parameter register, generates candidate coordinates through matrix transformation, and triggers the forward propagation computation of the multimodal large model. At this stage, the processor does not need to calculate the backpropagation gradient; instead, it directly extracts the probability distribution data from the model output layer Logits, calculates the probability difference between the first and second ranked parameters using the arithmetic logic unit (ALU) to determine the semantic conflict index. Based on the statistical characteristics (mean and variance) of this index, the processor performs advantage normalization and updates the search center parameters until it locks in the target geometric parameters that maximize the induction of spatial misjudgment in the model, thereby generating a geometric counterfactual reference sample.

[0088] Furthermore, the device implements a two-layer correction mechanism through the parallel computing capabilities of the processor. In the implicit correction stage, the processor loads the hidden state vectors corresponding to the original samples and counterfactual samples into a cache, performs vector subtraction and L2 norm normalization operations, calculates the geometric guidance features indicating the direction of spatial semantic correction, and superimposes them back into the original feature stream according to a preset strength coefficient. In the explicit correction stage, the processor calculates the Jensen-Shannon divergence between the original output distribution and the reference output distribution, uses this divergence value to determine dynamic weights through table lookup or function mapping, and performs linear interpolation to generate the final calibrated probability distribution. This calibrated distribution is finally decoded by the processor into the corresponding text token sequence, which is output as spatial inference response data through the communication interface.

[0089] In practical deployment scenarios, this device can be implemented as a high-performance server cluster deployed in a cloud data center, utilizing distributed storage and computing technologies to handle large-scale concurrent requests; or it can be implemented as an intelligent terminal device (such as an autonomous driving domain controller or an intelligent robot main control module) equipped with a high-performance edge computing chip to meet the requirements of low-latency real-time spatial inference. Furthermore, the memory can be volatile memory (such as DRAM) or non-volatile memory (such as Flash, SSD), with its internal logic divided into an instruction storage area and a data storage area, used to store the binary code of the aforementioned algorithm and model weight parameters, respectively. Through the deep collaboration of the aforementioned hardware and software, this device can significantly improve the accuracy and robustness of multimodal large models in complex spatial relationship judgment tasks without changing the original model parameters, effectively solving the spatial illusion problem caused by geometric mismatch in position encoding in existing technologies.

[0090] Example 3: This application further provides a computer-readable storage medium that serves as a software carrier for the above-described technical solutions, storing computer programs or instruction sequences thereon. When these computer programs are read and executed by the processor of an electronic device, the electronic device can be caused to perform all the steps of the multimodal large model spatial inference data processing method based on geometric manifold adaptation as described in the foregoing method embodiments.

[0091] In terms of specific physical implementation, the computer-readable storage medium can take various forms of non-volatile or volatile tangible media. For example, it can include, but is not limited to, semiconductor memories such as electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), dynamic random access memory (DRAM), and flash memory; it can also include magnetic or electronic high-capacity storage devices such as hard disk drives (HDDs) and solid-state drives (SSDs); and it can also include optical storage media such as optical discs (CD-ROMs, DVD-ROMs), and Blu-ray discs. Furthermore, with the development of cloud computing technology, the storage medium can also be embodied as a virtualized storage resource pool in a cloud server array, or a cache medium transmitted over a network and cached on a local terminal. Regardless of the physical form, its core characteristic is the ability to persistently or temporarily record binary code, which can be recognized and decoded into specific control levels by the computer's instruction register.

[0092] From a logical structure perspective, the computer program stored in the computer-readable storage medium is not a messy stack of data, but rather modularly organized into functional units. Specifically, the computer program includes a code segment for constructing a geometric manifold space, which encapsulates the mathematical definition of a positional encoding function, capable of mapping an input visual feature sequence into feature embeddings with a specific topological structure; a core algorithm module for performing manifold search operations, which integrates zero-order optimization logic, capable of controlling the processor to perform contrastive sampling to generate a symmetric noise set, calculating the statistical gradient of the semantic conflict index, and iteratively updating the search parameters accordingly to lock in the optimal geometric interference term; and code logic for generating geometric counterfactual reference samples, as well as an operational module for performing two-layer correction. The implicit correction code segment is responsible for performing vector subtraction and L2 norm normalization operations to generate geometrically guided features, while the explicit correction code segment is responsible for calculating the Jensen-Shannon divergence and performing dynamic weighted interpolation operations.

[0093] In practical applications, this computer-readable storage medium can be distributed as a standalone software product (such as a Software Development Kit (SDK) or Over-The-Air (OTA) firmware update package), or it can be pre-installed in the system partition of a multimodal large-model inference server. When the storage medium is connected to or integrated into an electronic device, the device's operating system reads the instructions from the medium through the file system, loads them into random access memory (RAM), and the central processing unit (CPU) or AI acceleration chip (such as a GPU / NPU) sequentially extracts the instructions from the pipeline. In this way, the storage medium transforms abstract mathematical algorithms into portable and reproducible industry standard components, enabling computing devices with different architectures to obtain the technical capability to eliminate the spatial illusion of multimodal large models by running the programs stored on it, thereby significantly improving the accuracy and reliability of spatial inference tasks.

[0094] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0095] The foregoing has shown and described the basic principles, main features, and advantages of this application. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this application. Various changes and modifications can be made to this application without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of the claims of this application.

Claims

1. A method for processing multimodal large model space inference data based on geometric manifold space, characterized in that, The method includes: Obtain the visual feature sequence corresponding to the image, and construct a geometric manifold space mapped to the visual feature sequence according to a preset position encoding function, wherein the geometric manifold space is a feature embedding space that characterizes the geometric topological structure of the position coordinates corresponding to the visual feature sequence; A manifold search operation is performed within the geometric manifold space. Candidate coordinates are generated by applying parameterized geometric transformations to the position coordinates. Target geometric parameters are determined based on the semantic conflict index calculated by the multimodal large model for the candidate coordinates. Geometric counterfactual reference samples are generated based on the target geometric parameters. The original sample data of the inference request to be processed and the geometric counterfactual reference sample are respectively input into the decoder of the multimodal large model, and the original hidden state data and reference hidden state data output by the decoder in the preset intermediate layer are extracted. Calculate the geometric guidance features between the original hidden state data and the reference hidden state data, and superimpose the geometric guidance features onto the original hidden state data to generate the corrected hidden state data; Decoding is performed based on the corrected hidden state data to generate an original output distribution, and a distribution difference metric between the original output distribution and the reference output distribution is calculated. The original output distribution is dynamically weighted and calibrated based on the distribution difference metric, and spatial reasoning response data corresponding to the reasoning request to be processed is generated based on the dynamically weighted and calibrated original output distribution. The reference output distribution is a probability distribution generated based on the geometric counterfactual reference sample.

2. The method according to claim 1, characterized in that, The manifold search operation performed within the geometric manifold space includes: Configure the rotation angle parameter of the preset position encoding function as an optimizable variable in the search space, and perform an initialization operation on the search center parameter of the optimizable variable; In response to entering the iterative search loop, a random noise sampling operation is performed to generate random noise data that follows a preset distribution, and the attenuation coefficient corresponding to the current iteration round is obtained; Based on the attenuation coefficient, a scaling operation is performed on the random noise data to generate scaled perturbation data; The scaling perturbation data is superimposed with the search center parameters to generate candidate angle parameters; Based on the candidate angle parameters, the parameterized geometric transformation is invoked to perform a coordinate transformation operation on the position coordinates, thereby generating the candidate coordinates; Obtain the semantic conflict index calculated for the candidate coordinates, and perform gradient estimation operation based on the semantic conflict index to generate statistical gradient data; The step size update operation is performed on the search center parameters based on the statistical gradient data, and the final search center parameters are output as the target geometric parameters when the preset convergence condition is met.

3. The method according to claim 2, characterized in that, The random noise sampling operation is performed to generate random noise data that follows a preset distribution, and the attenuation coefficient corresponding to the current iteration round is obtained; based on the attenuation coefficient, a scaling operation is performed on the random noise data to generate scaled perturbation data; The scaling perturbation data is superimposed with the search center parameters to generate candidate angle parameters, including: Call the random number generator to generate an original noise vector that follows a standard normal distribution; The original noise vector is inverted to generate the corresponding negative noise vector; The original noise vector is combined with the negative noise vector to construct a symmetric noise set as the random noise data. Each vector element in the symmetric noise set is multiplied by the attenuation coefficient to obtain a symmetric perturbation vector sequence, which is then used as the scaled perturbation data. Each vector in the symmetric perturbation vector sequence is added to the search center parameter to generate a pair of candidate angle parameters.

4. The method according to claim 2, characterized in that, The step of obtaining the semantic conflict index calculated for the candidate coordinates includes: The candidate coordinates are loaded into the input layer of the multimodal large model and the forward inference process is triggered to obtain the predicted probability distribution vector of the object to be processed generated by the output layer. Perform a descending sorting operation on the probability values ​​in the predicted probability distribution vector, and extract the first probability value with the first sorting position and the second probability value with the second sorting position. Perform a subtraction operation to calculate the difference between the first probability value and the second probability value, and mark the difference as the semantic conflict index.

5. The method according to claim 2, characterized in that, The step of performing gradient estimation based on the semantic conflict index to generate statistical gradient data includes: For the multiple candidate angle parameters generated in the current iteration round, obtain the corresponding multiple semantic conflict indicators; Perform statistical operations on the multiple semantic conflict indicators to calculate the mean parameter and variance parameter; Based on the mean parameter and the variance parameter, standardization processing is performed on each of the semantic conflict indicators to generate dominance value data; Obtain the scaling perturbation data corresponding to each of the aforementioned advantage value data, and calculate the weighted sum of the advantage value data and the scaling perturbation data, using the weighted sum as the statistical gradient data.

6. The method according to claim 1, characterized in that, The step of applying a parameterized geometric transformation to the position coordinates to generate candidate coordinates includes: Extract the horizontal component of the position coordinates from the geometric manifold space. With vertical component And obtain the horizontal component of the image center coordinates. With vertical component ; Based on the target geometric parameters Construct the rotation transformation matrix; The horizontal component Vertical component Horizontal components and vertical components Substitute the rotation transformation matrix into the matrix and perform matrix operations according to the following formula to generate the transformed coordinates. : 。 7. The method according to claim 1, characterized in that, The step of calculating the geometric guidance features between the original hidden state data and the reference hidden state data, and superimposing the geometric guidance features onto the original hidden state data to generate corrected hidden state data, includes: Calculate the original hidden state data by performing vector subtraction. With the reference hidden state data The difference vector; Perform an L2 norm operation on the difference vector to obtain the modulus value; Dividing the difference vector by the magnitude value yields the normalized geometric guide vector. The calculation process satisfies the formula: Obtain the preset correction intensity scalar Calculate the geometric guidance vector With the corrected intensity scalar The product of the original hidden state data is then combined with the product of the original hidden state data. Perform vector addition to generate the corrected hidden state data. .

8. The method according to claim 1, characterized in that, The step of dynamically weighting and calibrating the original output distribution based on the distribution divergence includes: The divergence calculation function is invoked to calculate the Jensen-Shannon divergence value between the original output distribution and the reference output distribution; The Jensen-Shannon divergence values ​​are input into a preset mapping function to generate dynamic weight coefficients. ; Based on the dynamic weighting coefficient The original output distribution is processed according to the following linear interpolation logic. With the reference output distribution Perform a weighted fusion operation to generate a calibrated distribution. : 。 9. A multimodal large-model spatial reasoning data processing device, characterized in that, The device includes: Memory, configured to store computer-executable instructions; A processor is configured to execute the computer-executable instructions, which, when executed, cause the apparatus to perform the steps of the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images