Visual feature representation learning method and system based on two-dimensional state space model

Abstract

The application discloses a visual feature representation learning method and system based on a two-dimensional state space model, and the method comprises the following steps: acquiring an input image; taking four vertices of the input image as starting points, and modeling a two-dimensional state space equation from four directions of the top left, the top right, the bottom right and the bottom left to construct a two-dimensional state space Mamba model; wherein, the modeling process of the two-dimensional state space equation is converted into modeling of two independent one-dimensional state space equations; and the two-dimensional state space Mamba model is used to perform a recognition and prediction task in the field of computer vision. The application uses the two-dimensional state space model to adaptively construct image feature representation, integrates the advantages of the Mamba model into visual processing, and meanwhile, local feature invariance is retained.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning technology, and in particular to a method and system for learning visual feature representations based on a two-dimensional state space model. Background Technology

[0002] Research based on state-space models (SSMs) has recently garnered significant attention in the field of Natural Language Processing (NLP). These works combine the advantages of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), achieving outstanding performance across various tasks due to their flexibility in handling multiple modalities and their efficient ability to process long sequences. Among them, Mamba, with its ability to handle temporal variations in input and its cleverly designed hardware acceleration algorithms, holds promise as a general-purpose framework to compete with transformers in NLP. Specifically, Mamba addresses the key issue of previous state-space models' insensitivity to different input samples by generating unique responses to different input samples. Simultaneously, it compensates for its processing speed limitations through underlying algorithms such as parallel scanning. Furthermore, compared to transformer-based models, Mamba has observed superior results in several benchmark tests.

[0003] In the field of computer vision, although architectures based on convolutional neural networks (CNNs) and visual transformers (ViTs) have long dominated, a number of works inspired by Mamba have attempted to apply Mamba to modeling visual data. These primarily focus on converting two-dimensional image data into one-dimensional sequences to accommodate Mamba's input format for subsequent processing. For example, Vision Mamba (Vim) employs a relatively straightforward approach: flattening the image row by row and then using a bidirectional modeling strategy to enhance the model's latent representational power. However, this simple flattening method significantly compromises the image's local invariance by stretching similar markers in the resulting one-dimensional sequence. Furthermore, LocalMamba, through strategies such as local scanning and optimal scan direction search, preserves some of the image's local invariance. Nevertheless, this approach still cannot fundamentally solve the problem of Mamba's incompatibility with modeling image data, which limits the model's ability to extract features from visual data. Summary of the Invention

[0004] The present invention aims to solve, at least to a certain extent, one of the technical problems in the related art.

[0005] To address this, this invention proposes a visual feature representation learning method based on a two-dimensional state-space model. Instead of simply converting image samples into one-dimensional sequences to fit the one-dimensional state-space Mamba model, this invention proposes Visual Two-Dimensional Mamba (V2M) as a solution. This extends the one-dimensional state-space equations in Mamba to a two-dimensional form, fundamentally adapting to the structure of image data, which uses two-dimensional state variables to describe each latent label. Furthermore, considering the non-sequential nature of image labels, this invention constructs the two-dimensional state equations starting from all four corners of the image. The proposed V2M method essentially preserves the local invariance of image representation while retaining the efficient modeling characteristics of the Mamba model.

[0006] Another objective of this invention is to propose a visual feature representation learning system based on a two-dimensional state space model.

[0007] To achieve the above objectives, this invention proposes a visual feature representation learning method based on a two-dimensional state-space model, comprising:

[0008] Obtain the input image;

[0009] Starting from the four vertices of the input image, a two-dimensional state-space equation is modeled from the four directions of upper left, upper right, lower right, and lower left to construct a two-dimensional state-space Mamba model; wherein, the modeling process of the two-dimensional state-space equation is transformed into the modeling of two independent one-dimensional state-space equations.

[0010] The two-dimensional state-space Mamba model is used to perform recognition and prediction tasks in the field of computer vision; wherein, the recognition and prediction tasks include image classification, object detection, semantic segmentation, and instance segmentation.

[0011] The visual feature representation learning method based on a two-dimensional state-space model in this invention may also have the following additional technical features:

[0012] In one embodiment of the present invention, the basis of the one-dimensional state-space equation is to map the one-dimensional input to the corresponding output using hidden state vectors, as shown below:

[0013] h′(t)=Ah(t)+Bx(t),

[0014] y(t) = Ch(t),

[0015] Where x(t) and y(t) represent the input and output vectors, respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices.

[0016] The formula for discretization using the zero-order preservation technique is as follows:

[0017]

[0018] Where Δ is the time scale parameter. and These are the discretized forms of A and B, respectively. The one-dimensional state-space equations in the discretized form are shown below:

[0019]

[0020] y t =Ch t .

[0021] Where, x t and y t h represents the discrete-form input and output vectors, respectively. t This represents the hidden state vector in discrete form.

[0022] In one embodiment of the present invention, for the input image, the one-dimensional state-space model is extended to a two-dimensional form, as follows:

[0023]

[0024] Among them, h i，j 1 and h i，j 2 These represent the hidden state vectors in the horizontal and vertical directions, respectively. The modeling process in the horizontal direction is as follows:

[0025]

[0026] Expanded into the following two parts:

[0027]

[0028] The first part is consistent with the form of the one-dimensional state-space equation, and the calculation result of the first part is denoted as x. i，j And substitute it into the formula in the second part:

[0029]

[0030] h i，j 2 The calculation process is only related to the hidden state vector in the vertical direction, as follows:

[0031]

[0032] In one embodiment of the present invention, the input images in four directions are rotated by different angles:

[0033] xi=rot(π / 2)·i(x), i=0, 1, 2.3,

[0034] The input images, after being rotated at different angles, are stitched together as shown below:

[0035] z=concat([x0, x1, x2, x3], dim=0).

[0036] In one embodiment of the present invention, the method further includes:

[0037] Obtain the stitching result of the input image after being rotated at different angles;

[0038] The splicing result is modeled using a two-dimensional state-space equation;

[0039] The modeled image features are aggregated based on the original position of the image before rotation.

[0040] To achieve the above objectives, another aspect of the present invention proposes a visual feature representation learning system based on a two-dimensional state-space model, comprising:

[0041] The image acquisition module is used to acquire the input image;

[0042] The spatial model construction module is used to take the four vertices of the input image as starting points and model the two-dimensional state space equations from the four directions of upper left, upper right, lower right, and lower left to construct a two-dimensional state space Mamba model; wherein, the modeling process of the two-dimensional state space equations is transformed into the modeling of two independent one-dimensional state space equations.

[0043] The task execution module is used to perform recognition and prediction tasks in the field of computer vision using the two-dimensional state space Mamba model; wherein, the recognition and prediction tasks include image classification, object detection, semantic segmentation, and instance segmentation.

[0044] The present invention provides a visual feature representation learning method and system based on a two-dimensional state-space model. This addresses the problem that existing methods for applying the Mamba model to the vision domain often compromise the modeling of local image features due to the local invariance of images and the temporal characteristics of the Mamba model. The present invention adaptively constructs image feature representations using a two-dimensional state-space model, incorporating the advantages of the Mamba model into visual processing while preserving local feature invariance.

[0045] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart of a visual feature representation learning method based on a two-dimensional state-space model according to an embodiment of the present invention;

[0048] Figure 2 This is an architecture diagram of a visual feature representation learning method based on a two-dimensional state space model according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram comparing the present invention with other metric learning methods according to embodiments of the present invention;

[0050] Figure 4 This is a structural diagram of a visual feature representation learning system based on a two-dimensional state-space model according to an embodiment of the present invention. Detailed Implementation

[0051] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0053] The following describes, with reference to the accompanying drawings, a visual feature representation learning method and system based on a two-dimensional state-space model according to embodiments of the present invention.

[0054] Figure 1 This is a flowchart of a visual feature representation learning method based on a two-dimensional state-space model according to an embodiment of the present invention, such as... Figure 1 As shown, it includes:

[0055] S1, Obtain the input image;

[0056] S2, taking the four vertices of the input image as starting points, models the two-dimensional state space equations from the four directions of upper left, upper right, lower right, and lower left to construct a two-dimensional state space Mamba model; wherein, the modeling process of the two-dimensional state space equations is transformed into the modeling of two independent one-dimensional state space equations.

[0057] S3 utilizes a two-dimensional state-space Mamba model to perform recognition and prediction tasks in the field of computer vision; wherein, the recognition and prediction tasks include image classification, object detection, semantic segmentation, and instance segmentation.

[0058] Figure 2 This is an architectural diagram of the present invention, such as Figure 2 As shown, this invention uses 2D SSM for feature encoding starting from four directions (top left, top right, bottom right, and bottom left).

[0059] It is understood that the present invention acquires the input image for subsequent feature processing.

[0060] In one embodiment of the present invention, the one-dimensional state-space equation Mamba model is based on mapping one-dimensional inputs to their corresponding outputs using hidden state vectors, as shown below:

[0061] h′(t)=Ah(t)+Bx(t),

[0062] y(t) = Ch(t),

[0063] Where x(t) and y(t) represent the input and output vectors, respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices.

[0064] The above is a continuous-form state-space equation. However, in most applications of artificial intelligence, the input is discrete. The formula for discretization using the zero-order preservation technique is as follows:

[0065]

[0066] Where Δ is the time scale parameter. and These are the discretized forms of A and B, respectively. The one-dimensional state-space equations in the discretized form are shown below:

[0067]

[0068] yt=Ch t .

[0069] Where, x t and y t h represents the discrete-form input and output vectors, respectively. t This represents the hidden state vector in discrete form.

[0070] In one embodiment of the present invention, for image samples, extending the state-space model to a two-dimensional form constitutes a more fundamental feature extraction method, as shown below:

[0071]

[0072] Among them, h i，j 1 and h i，j 2 These represent the hidden state vectors in the horizontal and vertical directions, respectively. The meanings of other parameters are consistent with the one-dimensional state space equation. In the specific implementation, this invention uses the modeling process in the horizontal direction as an example for illustration, as detailed below:

[0073]

[0074] The above formula can be expanded into the following two parts:

[0075]

[0076] The first part is consistent with the form of the one-dimensional state-space equation, and the calculation result of the first part is denoted as x. i,j And substitute it into the formula in the second part:

[0077]

[0078] To simplify the calculation process, this invention will use h i，j 2 The calculation process is only related to the hidden state vector in the vertical direction, as follows:

[0079]

[0080] The above form is also related to one-dimensional state-space equations. Therefore, this invention transforms the modeling process of two-dimensional state-space equations into the modeling of two independent one-dimensional state-space equations, the difference being the setting of parameters (A1, B1 for the former, and A2, I for the latter).

[0081] The embodiments of this invention use two-dimensional state-space equations to model image features, thereby establishing a more reasonable and ingenious application of Mamba in the field of computer vision.

[0082] Furthermore, considering the non-temporal nature of image data, this invention uses the four corners of the image as starting positions to calculate the two-dimensional state-space equations. In implementation, this invention rotates the input by different angles:

[0083] x i =rot (π / 2)·i (x), i = 0.1, 2, 3,

[0084] Then, the present invention splices the inputs after being rotated at different angles, as shown below:

[0085] z=concat([x0, x1, x2, x3], dim=0).

[0086] The spliced ​​results can be used to model the two-dimensional state-space equations, and the modeled features will be aggregated based on their original positions before rotation.

[0087] Preferably, the present invention uses an iterative method to model the two-dimensional state-space equations, with the number of iterations set to 24.

[0088] This invention constructs a two-dimensional visual Mamba module, starting from the four vertices of the image, and performs two-dimensional state space equation modeling and calculation respectively, thereby further enhancing the ability to understand the image.

[0089] Furthermore, after the model is built, the constructed two-dimensional Mamba model can be used to perform upstream recognition tasks and downstream dense prediction tasks in the field of computer vision, including image classification, object detection, semantic segmentation and instance segmentation, so as to comprehensively evaluate the effect of this method.

[0090] Furthermore, Figure 3 This is a schematic diagram comparing the present invention with other metric learning methods. Figure 3 In (a), Mamba encodes text information using blocks based on a one-dimensional state-space model after word segmentation. Figure 3 In (b), Visual Mamba (Vim) employs a method that directly flattens the input image and then processes it through a one-dimensional state space model block similar to Mamba. Figure 3 (c) in this paper shows that the V2M framework proposed in this invention introduces a two-dimensional state space model in the coding block without performing a flattening operation, which is more suitable for image processing.

[0091] This invention employs a two-dimensional state-space equation to model images, thereby preserving the inherent local invariance priors in images. Considering the non-temporal nature of the input image, this invention simultaneously models the two-dimensional state-space equation along four directions to enhance the model's representational power. The effectiveness of the framework was evaluated in classification and various downstream tasks, such as object detection, instance segmentation, and semantic segmentation. The results demonstrate the superiority of the two-dimensional visual Mamba model framework in classification and transfer performance.

[0092] The visual feature representation learning method based on a two-dimensional state-space model according to embodiments of the present invention addresses the problem that, when applying the Mamba model to the vision domain, simple image flattening operations often compromise the modeling of local image features due to the local invariance of images and the temporal characteristics of the Mamba model. Specifically, it adaptively constructs image feature representations using a two-dimensional state-space model, incorporating the advantages of the Mamba model into visual processing while preserving local feature invariance.

[0093] To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides visual feature representation learning 10 based on a two-dimensional state space model, including:

[0094] Image acquisition module 100 is used to acquire input images;

[0095] The spatial model construction module 200 is used to take the four vertices of the input image as starting points and model the two-dimensional state space equations from the four directions of upper left, upper right, lower right and lower left to construct a two-dimensional state space Mamba model; wherein, the modeling process of the two-dimensional state space equations is transformed into the modeling of two independent one-dimensional state space equations.

[0096] The task execution module 300 is used to perform recognition and prediction tasks in the field of computer vision using the two-dimensional state space Mamba model; wherein, the recognition and prediction tasks include image classification, object detection, semantic segmentation and instance segmentation.

[0097] For example, the basis of the one-dimensional state-space equation is to map the one-dimensional input to the corresponding output using hidden state vectors, as shown below:

[0098] h′(t)=Ah(t)+Bx(t),

[0099] y(t) = Ch(t),

[0100] Where x(t) and y(t) represent the input and output vectors, respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices.

[0101] The formula for discretization using the zero-order preservation technique is as follows:

[0102]

[0103] Where Δ is the time scale parameter. and These are the discretized forms of A and B, respectively. The one-dimensional state-space equations in the discretized form are shown below:

[0104]

[0105] yt=Ch t .

[0106] Where, x t and y t h represents the discrete-form input and output vectors, respectively. t This represents the hidden state vector in discrete form.

[0107] For example, for the input image, the one-dimensional state-space model is extended to a two-dimensional form as follows:

[0108]

[0109] Among them, h i，j 1 and h i，j 2 These represent the hidden state vectors in the horizontal and vertical directions, respectively. The modeling process in the horizontal direction is as follows:

[0110]

[0111] Expanded into the following two parts:

[0112]

[0113] The first part is consistent with the form of the one-dimensional state-space equation, and the calculation result of the first part is denoted as x. i,j And substitute it into the formula in the second part:

[0114]

[0115] h i，j 2 The calculation process is only related to the hidden state vector in the vertical direction, as follows:

[0116]

[0117] For example, the input images in four directions are rotated by different angles:

[0118] x i =rot (π / 2)·i (x), i = 0, 1, 2, 3,

[0119] The input images, after being rotated at different angles, are stitched together as shown below:

[0120] z=concat([x0, x1, x2, x3], dim=0).

[0121] Exemplarily, the system is also used for:

[0122] Obtain the stitching result of the input image after being rotated at different angles;

[0123] The splicing result is modeled using a two-dimensional state-space equation;

[0124] The modeled image features are aggregated based on the original position of the image before rotation.

[0125] The visual feature representation learning system based on a two-dimensional state-space model according to embodiments of the present invention addresses the problem that, when applying the Mamba model to the vision domain, simple image flattening operations often compromise the modeling of local image features due to the local invariance of images and the temporal characteristics of the Mamba model. Specifically, it adaptively constructs image feature representations using a two-dimensional state-space model, incorporating the advantages of the Mamba model into visual processing while preserving local feature invariance.

[0126] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A visual feature representation learning method based on a two-dimensional state-space model, characterized in that, include: Obtain the input image; Starting from the four vertices of the input image, a two-dimensional state-space equation is modeled from the four directions of upper left, upper right, lower right, and lower left to construct a two-dimensional state-space Mamba model; wherein, the modeling process of the two-dimensional state-space equation is transformed into the modeling of two independent one-dimensional state-space equations. For the input image, the one-dimensional state-space model is extended to a two-dimensional form, as follows: in, and These represent the hidden state vectors in the horizontal and vertical directions, respectively. The modeling process in the horizontal direction is as follows: Expanded into the following two parts: The first part is consistent with the form of the one-dimensional state-space equation, and the calculation result of the first part is denoted as... And substitute it into the formula in the second part: Will The calculation process is only related to the hidden state vector in the vertical direction, as follows: ； The two-dimensional state-space Mamba model is used to perform recognition and prediction tasks in the field of computer vision; wherein, the recognition and prediction tasks include image classification, object detection, semantic segmentation, and instance segmentation.

2. The method according to claim 1, characterized in that, The basis of the one-dimensional state-space equation is to use hidden state vectors to map the one-dimensional input to the corresponding output, as shown below: in and These represent the input and output vectors, respectively. Represents the hidden state vector. This represents the evolution parameter matrix of the one-dimensional state-space equation. and Represents the projection parameter matrix; The formula for discretization using the zero-order preservation technique is as follows: in, It is a time scale parameter. and They are and The discretized form of the state-space equations is shown below: in, and These represent the discrete-form input and output vectors, respectively. This represents the hidden state vector in discrete form.

3. The method according to claim 1, characterized in that, Rotate the input images in four directions by different angles: The input images, after being rotated at different angles, are stitched together as shown below: 。 4. The method according to claim 3, characterized in that, The method further includes: Obtain the stitching result of the input image after being rotated at different angles; The splicing result is modeled using a two-dimensional state-space equation; The modeled image features are aggregated based on the original position of the image before rotation.

5. A visual feature representation learning system based on a two-dimensional state-space model, characterized in that, include: The image acquisition module is used to acquire the input image; The spatial model construction module is used to take the four vertices of the input image as starting points and model the two-dimensional state space equations from the four directions of upper left, upper right, lower right, and lower left to construct a two-dimensional state space Mamba model; wherein, the modeling process of the two-dimensional state space equations is transformed into the modeling of two independent one-dimensional state space equations. For the input image, the one-dimensional state-space model is extended to a two-dimensional form, as follows: in, and These represent the hidden state vectors in the horizontal and vertical directions, respectively. The modeling process in the horizontal direction is as follows: Expanded into the following two parts: The first part is consistent with the form of the one-dimensional state-space equation, and the calculation result of the first part is denoted as... And substitute it into the formula in the second part: Will The calculation process is only related to the hidden state vector in the vertical direction, as follows: ； The task execution module is used to perform recognition and prediction tasks in the field of computer vision using the two-dimensional state space Mamba model; wherein, the recognition and prediction tasks include image classification, object detection, semantic segmentation, and instance segmentation.

6. The system according to claim 5, characterized in that, The basis of the one-dimensional state-space equation is to use hidden state vectors to map the one-dimensional input to the corresponding output, as shown below: in and These represent the input and output vectors, respectively. Represents the hidden state vector. This represents the evolution parameter matrix of the one-dimensional state-space equation. and Represents the projection parameter matrix; The formula for discretization using the zero-order preservation technique is as follows: in, It is a time scale parameter. and They are and The discretized form of the state-space equations is shown below: in, and These represent the discrete-form input and output vectors, respectively. This represents the hidden state vector in discrete form.

7. The system according to claim 5, characterized in that, Rotate the input images in four directions by different angles: The input images, after being rotated at different angles, are stitched together as shown below: 。 8. The system according to claim 7, characterized in that, The system is also used for: Obtain the stitching result of the input image after being rotated at different angles; The splicing result is modeled using a two-dimensional state-space equation; The modeled image features are aggregated based on the original position of the image before rotation.

Images