Method and apparatus for processing image data based on Mamba model

Abstract

A method and an apparatus for processing image data based on a Mama model are provided. The method comprises the following steps: receiving a data sequence corresponding to image data; processing the data sequence by using a Mamba model to obtain a data sequence processing result; and outputting a classification result corresponding to the image data based on the data sequence processing result. The Mama model comprises a Mama trunk block for processing a data sequence based on a selective state space model; and the plurality of calculation blocks are positioned behind the Mamba main block. Each of the plurality of calculation blocks includes: a global scoring unit performing a global evaluation to obtain an importance score; a context holding unit that generates relative position information of the token; and a slow-fast scanning unit performing a first selective state space update and a second selective state space update. Therefore, the Mama model can significantly reduce the number of floating point operations per second in the reasoning stage on the premise of not sacrificing the precision basically.

Description

Technical Field

[0001] This disclosure relates to deep learning, and more specifically, to methods and apparatus for processing image data based on the Mamba model. Background Technology

[0002] In recent years, the Mamba model, based on the sequence-state space modeling concept, has demonstrated excellent efficiency and accuracy in various scenarios, including vision and language processing. Compared to traditional architectures that rely on global attention, Mamba replaces quadratic complexity attention operations with linear-time selective scanning, giving it a natural advantage in long sequence modeling, streaming processing, and edge deployment.

[0003] However, as the scale of tasks and the complexity of data increase, Mamba in actual engineering still faces the problem of a large number of parameters and memory usage that cannot be ignored: on the one hand, the convolution kernels and selection gating involved in state updates will bring large intermediate activations; on the other hand, the hidden states passed across layers are highly sensitive to the numerical range, and slight improper quantization or pruning may cause "state drift", resulting in the destruction of long-range dependencies, which manifests as inference instability and a sharp drop in accuracy.

[0004] Furthermore, existing research on model compression largely focuses on quantization and pruning in dense architectures such as Transformers. For example, quantization scales are stabilized through equivalent transformations of layer normalization, or mixed-precision strategies are used to allocate bit widths by layer to balance accuracy and computational cost. These approaches achieve considerable resource savings in general-purpose language models, but they often rely on the assumption of "independent layers - independent errors," assuming that pruning or quantization errors will not be amplified in the temporal dimension. For Mamba, which has explicit state recursion and selection gating, these assumptions are not entirely valid: quantization noise accumulates along the scan direction and alters the gating distribution; structural sparsity resulting from pruning may disrupt relative order information and hidden state alignment, thus affecting the ability to reconstruct the global context. Simply applying compression paradigms for dense architectures can easily lead to a phenomenon where inference is normal in the early stages but performance significantly degrades as the sequence lengthens.

[0005] Therefore, designing a compression and inference mechanism that can significantly reduce the number of floating-point operations per second during the inference stage without sacrificing accuracy, based on the structural characteristics of Mamba, has become the key to promoting the stable operation of this type of model on lightweight devices. Summary of the Invention

[0006] The purpose of this disclosure is to provide a method and apparatus for processing image data based on the Mamba model that can significantly reduce the number of floating-point operations per second during the inference phase without substantially sacrificing accuracy.

[0007] According to one aspect of an embodiment of this disclosure, a method for processing image data based on a Mamba model is provided. The method includes: receiving a data sequence corresponding to image data, the data sequence including class labels and multiple tokens; processing the data sequence using a Mamba model to obtain a data sequence processing result; and outputting a classification result corresponding to the image data based on the data sequence processing result, wherein the Mamba model includes: a Mamba backbone block configured to process the data sequence based on a selective state-space model and output the processed data sequence; and multiple computational blocks connected in series and located after the Mamba backbone block, wherein each of the multiple computational blocks includes: a global scoring unit configured to: receive the processed data sequence or updated data. The sequence is used as the input sequence. A global evaluation is performed on the input sequence to obtain an importance score that has no positional bias over the entire sequence, and important tokens are selected from the plurality of tokens based on the importance score. The context preservation unit is configured to generate relative position information of the tokens in the input sequence. The slow-fast scan unit is configured to perform a first selective state space update on all tokens in the input sequence to obtain a first update result, perform a second selective state space update on important tokens in the input sequence to obtain a second update result, merge the second update result into the first update result based on the relative position information to obtain an updated data sequence, and output the updated data sequence.

[0008] Optionally, the global scoring unit performs global evaluation using global information interaction attention, the slow-fast scanning unit performs first selective state space update and second selective state space update using dynamic gaze-browsing scan, and the context preservation unit uses context position embedding to generate relative position information.

[0009] Optionally, the global scoring unit is configured to: strip class tags from the input sequence; and perform a global lightweight aggregation evaluation on all tokens of the stripped input sequence based on class tags in a query-based dimensionality reduction and key-based dimensionality reduction-up manner to obtain a position-bias-free importance score for each token.

[0010] Optionally, when the computation block is immediately following the Mamba backbone block, the global scoring unit of the computation block is configured to receive the processed data sequence from the Mamba backbone block as the input sequence of the computation block. When the computation block is not immediately following the Mamba backbone block, the global scoring unit of the computation block is configured to receive the updated data sequence from the previous computation block as the input sequence of the computation block. When the computation block is the last computation block among the plurality of computation blocks, the slow-fast scan unit of the computation block is configured to output the updated data sequence as the data sequence processing result. And when the computation block is not the last computation block among the plurality of computation blocks, the slow-fast scan unit of the computation block is configured to output the updated data sequence to the next computation block among the plurality of computation blocks.

[0011] Optionally, important tokens are propagated between adjacent computation blocks in a manner that excludes consistency.

[0012] Optionally, the slow-fast scan unit is configured to: perform a first selective state space update on all tokens in the input sequence in a first dimension to obtain a first update result; perform a second selective state space update on important tokens in the input sequence in a second dimension to obtain a second update result; merge the second update result into the first update result based on relative position information to obtain an updated data sequence; and output the updated data sequence, wherein the first dimension is smaller than the second dimension.

[0013] Optionally, the ratio of the first dimension to the second dimension is 1 / 8.

[0014] Optionally, the context-preserving unit is configured to generate an embedding matrix that encodes relative position information and provide the embedding matrix to the slow-fast scan unit.

[0015] According to one aspect of an embodiment of this disclosure, an apparatus for processing image data based on a Mamba model is provided. The apparatus includes: a receiving module configured to: receive a data sequence corresponding to image data, the data sequence including class tags and multiple tokens; a processing module configured to: process the data sequence using a Mamba model to obtain a data sequence processing result; and an output module configured to: output a classification result corresponding to the image data based on the data sequence processing result, wherein the Mamba model includes: a Mamba backbone block configured to process the data sequence based on a selective state-space model and output the processed data sequence; and multiple computation blocks connected in series and located after the Mamba backbone block, wherein each of the multiple computation blocks includes: a global scoring unit configured to: receive... The processed or updated data sequence is used as the input sequence. A global evaluation is performed on the input sequence to obtain an importance score that has no positional bias across the entire sequence, and important tokens are selected from the plurality of tokens based on the importance score. A context-preserving unit is configured to generate relative position information of the tokens in the input sequence. A slow-fast scanning unit is configured to perform a first selective state space update on all tokens in the input sequence to obtain a first update result, perform a second selective state space update on important tokens in the input sequence to obtain a second update result, fuse the second update result into the first update result based on the relative position information to obtain an updated data sequence, and output the updated data sequence.

[0016] According to one aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions stored in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the above-described method for processing image data based on the Mamba model.

[0017] According to one aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product includes computer instructions that, when executed by at least one processor, implement the above-described method for processing image data based on the Mamba model.

[0018] According to embodiments of this disclosure, the global scoring unit, context preservation unit, and slow-fast scanning unit work together to form a closed loop of "scoring-computation-position" and achieve an integrated solution of "global evaluation + slow-fast collaboration + position fallback". The Mamba model, while maintaining the overall sequence length and processing paradigm, significantly compresses the computational load and memory usage of Mamba-like models during the inference phase by triggering high-dimensional computation and state updates only at a few key locations. This is achieved in conjunction with the extremely low additional computation (approximately 1%~2.3% overhead) introduced by GIIA and CPE, while stabilizing accuracy loss within a very small range. The method and apparatus for processing image data based on the Mamba model according to embodiments of this disclosure are applicable to various backbone and multi-task scenarios and have good engineering portability and scalability, achieving a synchronous reduction in end-to-end latency and peak memory usage, and maintaining accuracy close to dense baselines in tasks such as classification, detection, and semantic segmentation. Therefore, the method and apparatus for processing image data based on the Mamba model according to embodiments of this disclosure can significantly reduce FLOPs during the inference phase and reduce peak memory usage and end-to-end latency with minimal loss of accuracy, thereby reducing computational and memory overhead while maintaining linear timing advantages. The method and apparatus for processing image data based on the Mamba model according to embodiments of this disclosure are particularly suitable for stable deployment in low-power scenarios such as resource-constrained edge computing and online streaming environments. Attached Figure Description

[0019] The above and / or other aspects of this disclosure will become clearer and more readily understood from the following detailed description taken in conjunction with the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a method for processing image data based on the Mamba model according to an embodiment of the present disclosure.

[0021] Figure 2 This is a block diagram illustrating a Mamba model according to an embodiment of the present disclosure.

[0022] Figure 3 This is a block diagram illustrating a computation block according to an embodiment of the present disclosure.

[0023] Figure 4 This is a block diagram illustrating an apparatus for processing image data based on the Mamba model according to an embodiment of the present disclosure.

[0024] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0025] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for the sake of clarity and conciseness, descriptions of features known upon understanding this disclosure may be omitted.

[0026] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0027] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Therefore, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0028] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the features, quantities, operations, components, elements, and / or combinations thereof stated therein, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as understood based on the disclosure of this application and as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having the same meaning as they have in the context of the relevant art and in the disclosure of this application, and shall not be interpreted ideally or overly formally. The use of the term “may” herein with respect to examples or embodiments (e.g., regarding what an example or embodiment may include or implement) indicates the existence of at least one example or embodiment that includes or implements such a feature, while not all examples are limited thereto.

[0030] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that reappear in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments can be combined in any way, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve the technical effects described or not described in this disclosure but which can be determined based on the complete technical solution described above. The terminology used in this disclosure is explained below.

[0031] It should be noted that, where there is no conflict between the various embodiments, these embodiments and their features can be combined with each other.

[0032] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0033] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0034] Figure 1 This is a flowchart illustrating a method for processing image data based on the Mamba model according to an embodiment of the present disclosure.

[0035] Reference Figure 1 The method for processing image data based on the Mamba model according to embodiments of the present disclosure may include steps S100 to S300.

[0036] In step S100, a data sequence corresponding to the image data is received. According to embodiments of this disclosure, the data sequence may include class tags and multiple tokens. In one example, a token (or tile token) corresponding to each patch and a class tag (or class token) responsible for the final classification / global semantics of the entire image can be obtained by segmenting the image (or image data) into multiple patches, and the data sequence corresponding to the image data is obtained by arranging the multiple tokens and class tags into a linear sequence. In one example, after step S100, the method for processing image data based on the Mamba model according to embodiments of this disclosure may further include converting the received data sequence into a representation that can be processed by the Mamba model.

[0037] In step S200, the data sequence is processed using a Mamba model to obtain a data sequence processing result. The Mamba model is a selective state-space model. In one example, the data sequence obtained in step S100 can be input into the Mamba model, causing the Mamba model to output a data sequence processing result. The method for processing image data based on the Mamba model according to embodiments of this disclosure is geared towards a Mamba model with selective scanning and explicit latent state propagation characteristics, which will be referred to below. Figure 2 Describe the Mamba model in detail.

[0038] In step S300, based on the data sequence processing results, a classification result corresponding to the image data is output. However, it should be understood that this disclosure is not limited thereto. For example, in one embodiment, based on the data sequence processing results, other prediction results (e.g., detection results or semantic segmentation results) corresponding to the image data may also be output.

[0039] Figure 2 This is a block diagram illustrating a Mamba model 100 according to an embodiment of the present disclosure.

[0040] Reference Figure 2 The Mamba model 100 includes Mamba backbone blocks 110 and multiple compute blocks 120. In this disclosure, the Mamba backbone blocks may also be referred to as the Mamba backbone layer, and the compute blocks may also be referred to as compute layers.

[0041] According to embodiments of this disclosure, Mamba backbone block 110 is configured to process a data sequence based on a selective state-space model and output the processed data sequence. According to embodiments of this disclosure, a plurality of computation blocks 120 are connected in series with each other and are located after the Mamba backbone block. For example, the plurality of computation blocks 120 may be connected end-to-end with each other, and the first computation block 120 of the plurality of computation blocks 120 may be connected to (e.g., directly connected to) the output of Mamba backbone block 110. According to embodiments of this disclosure, each of the plurality of computation blocks 120 may have the same configuration. Referring below... Figure 3 A detailed description of one of the multiple computation blocks 120.

[0042] Figure 3 This is a block diagram illustrating a computation block according to an embodiment of the present disclosure.

[0043] Reference Figure 3 According to embodiments of the present disclosure, the computation block 120 may include a global scoring unit 121, a context preservation unit 122, and a slow / fast scanning unit 123.

[0044] According to embodiments of this disclosure, the global scoring unit 121 is configured to receive a processed or updated data sequence as an input sequence. In one example, the global scoring unit 121 may use Global Information Interaction Attention (GIIA) to perform a global evaluation. Furthermore, in one example, when computation block 120 is a computation block immediately following Mamba backbone block 110, the global scoring unit 121 of that computation block 120 may be configured to receive a processed data sequence from Mamba backbone block 110 as an input sequence for that computation block 120; in another example, when computation block 120 is not a computation block immediately following Mamba backbone block 110, the global scoring unit 121 of that computation block 120 may be configured to receive an updated data sequence from the preceding computation block as an input sequence for that computation block 120.

[0045] According to embodiments of this disclosure, the global scoring unit 121 is further configured to perform a global evaluation on the input sequence to obtain an importance score with no positional bias over the entire sequence. For example, the global evaluation can be a one-time lightweight aggregation reference. In one example, the global scoring unit 121 can be configured to strip (or decouple) class tags from the input sequence and perform a global lightweight aggregation evaluation on all tokens of the stripped input sequence based on the class tags in a query dimensionality reduction and key-value dimensionality reduction-up manner to obtain an importance score with no positional bias for each token. That is, unlike the conventional approach in the prior art of directly embedding class tags into long sequences and accumulating information mainly through local neighborhood interactions, the global scoring unit 121 according to embodiments of this disclosure can actively decouple class tags from the sequential update path (i.e., strip the local temporal dependencies of class tags), and apply a very lightweight global aggregation only at the output of each block (e.g., Mamba backbone block 110 and / or computation block 120), so that the class tags can still "look down" on the entire sequence and form an importance score with no positional bias for each token without participating in one-dimensional recursion. This position-independent scoring is particularly crucial for long sequences because it avoids the source of bias where midpoints or nearest neighbors are systematically overestimated and far-ends are consistently underestimated. At the implementation level, the global scoring unit 121 according to embodiments of this disclosure can compress computation and memory usage through query dimensionality reduction and key-value down-up-down projection, maintaining the same linear time advantage as the Mamba backbone 110 without introducing destructive coupling. In engineering, the global scoring unit 121 according to embodiments of this disclosure can also be combined with a slight normalization strategy and temperature smoothing to make the importance distributions between different samples and batches comparable, thereby stabilizing subsequent dynamic selection.

[0046] According to embodiments of this disclosure, the global scoring unit 121 is further configured to select important tokens from multiple tokens based on importance scores. In one example, important tokens can be propagated between adjacent computation blocks 120 in a manner that excludes consistency, so that multiple computation blocks 120 have a stable attention trajectory and suppress jitter. That is, the set of important tokens (or high-scoring tokens) selected by the global scoring unit 121 according to embodiments of this disclosure is not independent of the layer, but is propagated along the network layers (blocks) in a manner that excludes consistency: the candidate set of the later layer is preferentially filtered in the key positions that have been passed in the previous layer, and only small additions and deletions are allowed when there is a significant change in the new context. This can significantly reduce the selection jitter between layers, ensure that the key positions that are retained still move along a clear attention trajectory after being stacked in depth, and avoid the oscillation of "the previous layer judged as key and the next layer immediately discarded". With optional upper and lower bound constraints, the Mamba model 100 according to embodiments of this disclosure can automatically shrink or expand the size of the key subset on samples of different complexity, so that the computational budget matches the sample difficulty.

[0047] According to embodiments of this disclosure, the context preservation unit 122 is configured to generate relative position information of tokens in the input sequence. In one example, the context preservation unit 122 may use Contextual Position Embedding (CPE) to generate the relative position information. Therefore, the Mamba model 100 (e.g., the slow-fast scan unit 123) can still retrieve relative position information from the complete sequence after dynamic selection, and the context preservation unit 122 can avoid order misalignment, semantic drift, and context misreading caused by dynamic selection altering the sequence morphology (e.g., order changes), thereby maintaining original context consistency and stabilizing context relationships under dynamic structures. Furthermore, the context preservation unit 122 can achieve cross-layer consistent usage to reduce positional jitter in the depth direction.

[0048] As described below, dynamic structures inevitably alter the sequence form "seen" by higher-dimensional branches. Without explicit handling of positional relationships, subsequent higher-dimensional processing can easily misinterpret the context, manifesting as insensitivity to distance constraints and overconfidence in boundary regions. To address this, in one example, the context-preserving unit 122 can be configured to generate an embedding matrix encoding relative positional information and provide this matrix to the slow-fast scanning unit 123. Thus, as described below, relative positional information is shared between the low-dimensional browsing and high-dimensional gaze paths. Intuitively, CPE is equivalent to attaching a "relative coordinate map" to each location, which, regardless of whether it is selected as a key, can be supplemented with relevant contextual relationships as additional terms during higher-dimensional computation. This sharing mechanism ensures that both paths reference the same set of relative relationship representations, thereby avoiding semantic gaps where "what is considered close in low-dimensional space is judged as distant in high-dimensional space." In terms of implementation details, CPE also incorporates two stabilization measures: first, it applies slow updates or moving averages to the CPE parameters across layers to reduce the fluctuations in relative relationships at different depths; second, it applies neighborhood expansion to the extremely sparse key set so that high-dimensional branches can "touch" a few non-key positions that are strongly related to key positions when necessary, thus maintaining semantic continuity.

[0049] According to an embodiment of this disclosure, the slow-fast scanning unit 123 is configured to: perform a first selective state space update on all tokens in the input sequence to obtain a first update result, perform a second selective state space update on important tokens in the input sequence to obtain a second update result, fuse the second update result into the first update result based on relative position information to obtain an updated data sequence, and output the updated data sequence.

[0050] In one example, the slow-fast scan unit 123 can be configured to: perform a first selective state space update on all tokens in the input sequence in a first dimension to obtain a first update result; perform a second selective state space update on important tokens in the input sequence in a second dimension to obtain a second update result; fuse the second update result into the first update result based on relative position information to obtain an updated data sequence; and output the updated data sequence. In one example, the first dimension may be smaller than the second dimension. Therefore, the first selective state space update performed on all tokens in the input sequence can preserve low-dimensional paths for all tokens to maintain continuous context carrying, and the second selective state space update performed on important tokens in the input sequence can achieve fine-grained updates that provide high-dimensional paths only for important tokens, so that bit-by-bit fusion is provided at key positions where important tokens are located and only low-dimensional update results are preserved at positions where non-important tokens are located. That is, the slow-fast scan unit 123 can thereby shrink high-cost computations to key positions while maintaining linear complexity, and at the same time preserve the minimum context for all tokens with low-dimensional channels. In one example, normalization can be performed after the first selective state space update and the second selective state space update to complete scale alignment. In a preferred example, the ratio of the first dimension to the second dimension can be 1 / 8, thus achieving a better trade-off between accuracy and efficiency.

[0051] In one example, the slow-fast scanning unit 123 can use Dynamic Gaze-Browse Scanning (DGBS) to perform a first selective state space update and a second selective state space update. In this case, the path performing the first selective state space update can be called the (low-dimensional) browsing path, and the path performing the second selective state space update can be called the (high-dimensional) gaze path. That is, after entering DGBS, all tokens will first maintain global context connectivity at very low cost through a low-dimensional "browsing" path. This path plays the role of "minimum context carrying": it does not pursue point-by-point optimization, but ensures that no matter how the key subset evolves, the model always holds a continuous semantic context covering the entire sequence. On this basis, only a small number of tokens judged as key by the global scoring unit 121 will be activated to the high-dimensional "gaze" path to perform a more refined selective state space update. The two are additively fused bit-by-bit at key positions, and the low-dimensional results are directly used at non-key positions, and then uniformly enter the normalization step to complete the scale alignment. Unlike traditional pruning that directly discards non-critical tokens, DGBS aims to compensate rather than discard them: low-dimensional browsing provides a "safety net" for the entire sequence, ensuring that even if a critical set is misjudged at a single layer, it won't create irreparable semantic gaps in the temporal chain. Furthermore, to further reduce latency, DGBS employs a grouped scheduling strategy: during inference, samples with similar critical proportions within the same mini-batch are grouped together to minimize the overhead of sparse jumps in high-dimensional branches; simultaneously, a soft lower bound is set on the critical proportion to prevent all samples from falling into the low dimension and losing discriminative power in extreme cases. In practice, the low-dimensional width is typically chosen as a fixed fraction of the high-dimensional width (e.g., one-eighth), which is empirically optimal under constraints such as floating-point operations per second (FLOPs) that balance accuracy and throughput. When downstream scenarios are extremely sensitive to latency, the low-dimensional width can be further reduced and the critical proportion slightly increased to keep the end-to-end response time below the target threshold. It is worth emphasizing that DGBS does not change the overall sequence length or processing paradigm. All tokens are updated and participate in the computation of subsequent layers in at least one of the two paths. Therefore, there is no need to rewrite the upstream data alignment and the downstream loss interface, which has good system compatibility.

[0052] In one example, when computation block 120 is the last computation block among multiple computation blocks 120, the slow-fast scan unit 123 of computation block 120 can be configured to output the updated data sequence as the data sequence processing result; in another example, when computation block 120 is not the last computation block among multiple computation blocks 120, the slow-fast scan unit 123 of computation block 120 can be configured to output the updated data sequence to the next computation block among multiple computation blocks 120.

[0053] In one example, each of the global scoring unit 121, context-preserving unit 122, and slow-fast scanning unit 123 can be implemented by hardware, software (e.g., a software stack), or a combination thereof (e.g., a hardware-side microkernel for lightweight global aggregation and location compensation). Visualization results show that when GIIA, DGBS, and CPE operate simultaneously, the high-response region almost overlaps with the dense baseline; once GIIA is removed, importance assessment becomes significantly biased towards local neighborhoods; if only widening the low dimension replaces the high-dimensional gaze, the model's discrimination boundary becomes blurred and insensitive to fine-grained structures; removing CPE easily leads to misalignment in long-distance dependencies, manifested as significant attenuation at cross-segment junctions. These phenomena are consistent with offline metrics, indicating that CPE plays a crucial fallback role in ensuring the consistency of location semantics under dynamic structures.

[0054] In summary, according to the embodiments of this disclosure, the global scoring unit 121, the context preservation unit 122, and the slow-fast scanning unit 123 work together to form a "scoring-computation-position" closed loop and achieve an integrated solution of "global evaluation + slow-fast collaboration + position fallback". The Mamba model 100, while maintaining the overall sequence length and processing paradigm, significantly compresses the computational load and memory usage of Mamba-like models during the inference phase by triggering high-dimensional computation and state updates only at a few key locations, combined with the extremely low additional computation (approximately 1%~2.3% overhead) introduced by GIIA and CPE, while stabilizing accuracy loss within a very small range. The method for processing image data based on the Mamba model according to the embodiments of this disclosure is applicable to various backbone and multi-task scenarios and has good engineering portability and scalability, achieving a synchronous reduction in end-to-end latency and peak memory usage, and maintaining accuracy close to dense baselines in tasks such as classification, detection, and semantic segmentation. Therefore, the method for processing image data based on the Mamba model according to embodiments of this disclosure can significantly reduce FLOPs during the inference phase and reduce peak memory usage and end-to-end latency with minimal loss of accuracy, thereby reducing computational and memory overhead while maintaining linear timing advantages. The method for processing image data based on the Mamba model according to embodiments of this disclosure is particularly suitable for stable deployment in low-power scenarios such as resource-constrained edge computing and online streaming environments.

[0055] Figure 4 This is a block diagram illustrating an apparatus for processing image data based on the Mamba model according to an embodiment of the present disclosure.

[0056] Reference Figure 4 According to an embodiment of the present disclosure, a device 1000 for processing image data based on the Mamba model includes a receiving module 1100, a processing module 1200, and an output module 1300.

[0057] According to embodiments of this disclosure, the receiving module 1100 is configured to receive a data sequence corresponding to image data, the data sequence including a class tag and multiple tokens. The receiving module 1100 can perform the operation corresponding to step S100, therefore repeated descriptions will be omitted.

[0058] According to embodiments of this disclosure, processing module 1200 is configured to: use the Mamba model (e.g., refer to...) Figure 2 and Figure 3 The described Mamba model 100 processes the data sequence to obtain the data sequence processing result. The processing module 1200 can perform the operation corresponding to step S200, so repeated descriptions will be omitted.

[0059] According to embodiments of this disclosure, the output module 1300 is configured to output a classification result corresponding to the image data based on the data sequence processing result. The output module 1300 can perform the operation corresponding to step S300, therefore repeated descriptions will be omitted.

[0060] In summary, according to the embodiments of this disclosure, the global scoring unit 121, the context preservation unit 122, and the slow-fast scanning unit 123 work together to form a "scoring-computation-position" closed loop and achieve an integrated solution of "global evaluation + slow-fast collaboration + position fallback". The Mamba model 100, while maintaining the overall sequence length and processing paradigm, significantly compresses the computational load and memory usage of Mamba-like models during the inference phase by triggering high-dimensional computation and state updates only at a few key locations, combined with the extremely low additional computation (approximately 1%~2.3% overhead) introduced by GIIA and CPE, while stabilizing accuracy loss within a very small range. The device for processing image data based on the Mamba model according to the embodiments of this disclosure is suitable for various backbone and multi-task scenarios and has good engineering portability and scalability, achieving a synchronous reduction in end-to-end latency and peak memory usage, and maintaining accuracy close to dense baselines in tasks such as classification, detection, and semantic segmentation. Therefore, the Mamba-based image data processing device according to embodiments of this disclosure can significantly reduce FLOPs during the inference phase and reduce peak memory usage and end-to-end latency with minimal loss of accuracy, thereby reducing computational and memory overhead while maintaining linear timing advantages. The Mamba-based image data processing device according to embodiments of this disclosure is particularly suitable for stable deployment in low-power scenarios such as resource-constrained edge computing and online streaming environments.

[0061] According to embodiments of this disclosure, a computer-readable storage medium may also be provided, wherein when instructions stored in the computer-readable storage medium are executed by at least one processor, the at least one processor is caused to perform the above-described method for processing image data based on the Mamba model. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0062] According to embodiments of this disclosure, a computer program product may also be provided, including computer instructions that, when executed by at least one processor, implement the above-described method for processing image data based on the Mamba model.

[0063] According to embodiments of this disclosure, the global scoring unit 121, context preservation unit 122, and slow-fast scanning unit 123 work together to form a "scoring-computation-position" closed loop and achieve an integrated solution of "global evaluation + slow-fast collaboration + position fallback". The Mamba model 100, while maintaining the overall sequence length and processing paradigm, significantly compresses the computational load and memory usage of Mamba-like models during the inference phase by triggering high-dimensional computation and state updates only at a few key locations, combined with the extremely low additional computation (approximately 1%~2.3% overhead) introduced by GIIA and CPE, while stabilizing accuracy loss within a very small range. The method and apparatus for processing image data based on the Mamba model according to embodiments of this disclosure are applicable to various backbone and multi-task scenarios and have good engineering portability and scalability, achieving a synchronous reduction in end-to-end latency and peak memory usage, and maintaining accuracy close to dense baselines in tasks such as classification, detection, and semantic segmentation. Therefore, the method and apparatus for processing image data based on the Mamba model according to embodiments of this disclosure can significantly reduce FLOPs during the inference phase and reduce peak memory usage and end-to-end latency with minimal loss of accuracy, thereby reducing computational and memory overhead while maintaining linear timing advantages. The method and apparatus for processing image data based on the Mamba model according to embodiments of this disclosure are particularly suitable for stable deployment in low-power scenarios such as resource-constrained edge computing and online streaming environments.

[0064] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0065] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for processing image data based on the Mamba model, characterized in that, The method includes: Receive a data sequence corresponding to the image data, the data sequence including a class label and multiple tokens; The data sequence is processed using the Mamba model to obtain the data sequence processing results; and Based on the data sequence processing results, the classification results corresponding to the image data are output. The Mamba model includes: Mamba backbone blocks are configured to process data sequences based on a selective state-space model and output the processed data sequences; and Multiple computational blocks are connected in series and located after the Mamba backbone blocks. Each of the plurality of computational blocks includes: The global scoring unit is configured to: receive the processed or updated data sequence as the input sequence, perform a global evaluation on the input sequence to obtain an importance score that is unbiased across the entire sequence, and select important tokens from the plurality of tokens based on the importance score; The context preservation unit is configured to: generate relative position information of tokens in the input sequence; and The slow-fast scanning unit is configured to: perform a first selective state space update on all tokens in the input sequence to obtain a first update result; perform a second selective state space update on important tokens in the input sequence to obtain a second update result; merge the second update result into the first update result based on relative position information to obtain an updated data sequence; and output the updated data sequence.

2. The method according to claim 1, characterized in that, The global scoring unit performs global evaluation using global information interactive attention, the slow-fast scanning unit performs first and second selective state space updates using dynamic gaze-browsing scanning, and the context preservation unit uses context position embedding to generate relative position information.

3. The method according to claim 1, characterized in that, The global scoring unit is configured as follows: Remove class labels from the input sequence; and Global lightweight aggregation evaluation is performed on all tokens of the input sequence after class tags have been removed, using query dimensionality reduction and key-value dimensionality reduction-up, to obtain a position-bias-free importance score for each token.

4. The method according to claim 1, characterized in that, When one of the plurality of computation blocks is a computation block immediately following a Mamba backbone block, the global scoring unit of that computation block is configured to receive the processed data sequence from the Mamba backbone block as the input sequence for that computation block. When the computation block is not immediately following a Mamba backbone block, the global scoring unit of the computation block is configured to receive an updated data sequence from the preceding computation block as the input sequence for the computation block. When the computation block is the last of the plurality of computation blocks, the slow-fast scan unit of the computation block is configured to output the updated data sequence as the data sequence processing result, and When the computation block is not the last computation block among the plurality of computation blocks, the slow-fast scan unit of the computation block is configured to output the updated data sequence to the next computation block among the plurality of computation blocks.

5. The method according to claim 1, characterized in that, Important tokens are propagated between adjacent computation blocks in a manner that excludes consistency.

6. The method according to claim 1, characterized in that, The slow and fast scan units are configured as follows: Perform a first selective state space update on all tokens in the input sequence along the first dimension to obtain a first update result; Perform a second selective state space update on the important tokens in the input sequence in the second dimension to obtain a second update result; The second update result is merged into the first update result based on relative position information to obtain an updated data sequence; and Output the updated data sequence. The first dimension is smaller than the second dimension.

7. The method according to claim 6, characterized in that, The ratio of the first dimension to the second dimension is 1 / 8.

8. The method according to claim 1, characterized in that, The context-preserving unit is configured to generate an embedding matrix that encodes relative position information and provide the embedding matrix to the slow-fast scan unit.

9. A device for processing image data based on the Mamba model, characterized in that, The device includes: The receiving module is configured to receive a data sequence corresponding to the image data, the data sequence including a class label and multiple tokens; The processing module is configured to: process the data sequence using a Mamba model to obtain the data sequence processing result; and The output module is configured to output classification results corresponding to the image data based on the data sequence processing results. The Mamba model includes: Mamba backbone blocks are configured to process data sequences based on a selective state-space model and output the processed data sequences; and Multiple computational blocks are connected in series and located after the Mamba backbone blocks. Each of the plurality of computational blocks includes: The global scoring unit is configured to: receive the processed or updated data sequence as the input sequence, perform a global evaluation on the input sequence to obtain an importance score that is unbiased across the entire sequence, and select important tokens from the plurality of tokens based on the importance score; The context preservation unit is configured to: generate relative position information of tokens in the input sequence; and The slow-fast scanning unit is configured to: perform a first selective state space update on all tokens in the input sequence to obtain a first update result; perform a second selective state space update on important tokens in the input sequence to obtain a second update result; merge the second update result into the first update result based on relative position information to obtain an updated data sequence; and output the updated data sequence.

10. A computer-readable storage medium, characterized in that, When the instructions stored in the computer-readable storage medium are executed by at least one processor, the at least one processor is caused to perform the method for processing image data based on the Mamba model according to any one of claims 1 to 8.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, the method for processing image data based on the Mamba model according to any one of claims 1 to 8 is implemented.

Images