Visual feature representation learning method and system based on two-dimensional state space model

Abstract

The invention discloses a visual feature representation learning method and system based on a two-dimensional state space model. The method comprises the following steps: acquiring an input image; the method comprises the following steps: by taking four vertexes of an input image as starting points, performing modeling of a two-dimensional state space equation from four directions of upper left, upper right, lower right and lower left to construct a two-dimensional state space Mama model; wherein the modeling process of a two-dimensional state-space equation is converted into modeling of two independent one-dimensional state-space equations; and executing an identification prediction task in the field of computer vision by using the two-dimensional state space Mama model. According to the method, a two-dimensional state space model is used to adaptively construct image feature representation, the advantages of a Mama model are integrated into visual processing, and local feature invariance is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a visual feature representation learning method and system based on a two-dimensional state space model. Background Art

[0002] Research based on state-space models (SSMs) has recently received sustained attention in the field of natural language processing (NLP). These works combine the advantages of convolutional neural networks and recurrent neural networks, and have achieved outstanding performance in various tasks due to their flexibility in handling multiple modalities and their ability to efficiently process long sequences. Among them, Mamba is expected to compete with transformers as a general framework in the field of NLP due to its ability to handle the time-varying characteristics of inputs and its cleverly designed hardware acceleration algorithms. Specifically, Mamba solves the key problem that previous state-space models are insensitive to different inputs by generating unique responses to different input samples. At the same time, it makes up for the lack of processing speed through underlying algorithms such as parallel scanning. In addition, compared with transformer-based models, Mamba has observed superior results in several benchmarks.

[0003] In the field of computer vision, although architectures based on convolutional neural networks (CNNs) and visual transformers (ViTs) have been dominant, inspired by Mamba, a series of works have attempted to apply Mamba to the modeling of visual data. They mainly focus on converting 2D image data into 1D sequences to adapt to Mamba's input format for subsequent processing. For example, Vision Mamba (Vim) adopts a relatively straightforward approach by flattening the image row by row and then adopting a bidirectional modeling strategy to enhance the model's potential representation ability. However, this simple flattening method significantly destroys the local invariance of the image by stretching similar markers in the obtained 1D sequence. In addition, LocalMamba preserves the local invariance of the image to a certain extent through strategies such as local scanning and optimal scanning direction search. Nevertheless, this approach still cannot fundamentally solve the problem that Mamba is incompatible with modeling image data, which limits the model's ability to extract features from visual data. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art to at least a certain extent.

[0005] To this end, the present invention proposes a visual feature representation learning method based on a two-dimensional state space model. The present invention does not simply convert image samples into one-dimensional sequences to adapt to Mamba based on a one-dimensional state space model, but proposes Visual Two-Dimensional Mamba (V2M) as a solution to expand the one-dimensional state space equations in Mamba to a two-dimensional form, fundamentally adapting to the structure of image data, which uses two-dimensional state variables to describe each potential tag. In addition, considering the non-sequential nature of image tags, the present invention constructs a two-dimensional state equation starting from all four corners of the image. The V2M method proposed in the present invention essentially retains the local invariance of image representation while retaining the efficient modeling characteristics of the Mamba model.

[0006] Another object of the present invention is to propose a visual feature representation learning system based on a two-dimensional state space model.

[0007] To achieve the above object, the present invention proposes a visual feature representation learning method based on a two-dimensional state space model, comprising:

[0008] Get the input image;

[0009] Taking the four vertices of the input image as starting points, modeling of the two-dimensional state space equation is performed from the upper left, upper right, lower right and lower left directions to construct a two-dimensional state space Mamba model; wherein the modeling process of the two-dimensional state space equation is converted into the modeling of two independent one-dimensional state space equations;

[0010] The two-dimensional state space Mamba model is used to perform recognition prediction tasks in the field of computer vision; wherein the recognition prediction tasks include image classification, target detection, semantic segmentation and instance segmentation.

[0011] The visual feature representation learning method based on the two-dimensional state space model of the embodiment of the present invention may also have the following additional technical features:

[0012] In one embodiment of the present invention, the basis of the one-dimensional state space equation uses a hidden state vector to map a one-dimensional input to a corresponding output result as shown below:

[0013] h′(t)=Ah(t)+Bx(t),

[0014] y(t)=Ch(t),

[0015] Where x(t) and y(t) represent the input and output vectors respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices;

[0016] The formula for discretization using the zero-order hold technique is as follows:

[0017]

[0018] Where Δ is the time scale parameter, and are the discretized forms of A and B respectively. The one-dimensional state space equation form of the discrete form is as follows:

[0019]

[0020] y t =Ch t .

[0021] Among them, x t and t Represent the input and output vectors in discrete form, h t Represents the hidden state vector in discrete form.

[0022] In one embodiment of the present invention, for the input image, the one-dimensional state space model is expanded to a two-dimensional form as shown below:

[0023]

[0024] Among them, h i，j 1 and h i，j 2 Represent the hidden state vectors in the horizontal and vertical directions respectively. The modeling process in the horizontal direction is as follows:

[0025]

[0026] Expanded into the following two parts:

[0027]

[0028] The first part is consistent with the form of the one-dimensional state space equation. The result of the first part is recorded as x i，j ′ and substitute it into the formula in the latter part:

[0029]

[0030] h i，j 2 The calculation process of is only related to the hidden state vector in the vertical direction, as follows:

[0031]

[0032] In one embodiment of the present invention, the input images in four directions are rotated at different angles respectively:

[0033] xi=rot(π / 2)·i(x), i=0, 1, 2.3,

[0034] The input images rotated at different angles are stitched together as shown below:

[0035] z=concat([x0, x1, x2, x3], dim=0).

[0036] In one embodiment of the present invention, the method further includes:

[0037] Get the stitching results of the input images after being rotated at different angles;

[0038] Modeling the splicing result with a two-dimensional state space equation;

[0039] The modeled image features are aggregated according to their original positions before image rotation.

[0040] To achieve the above object, the present invention further provides a visual feature representation learning system based on a two-dimensional state space model, comprising:

[0041] An image acquisition module, used for acquiring an input image;

[0042] A spatial model building module is used to take the four vertices of the input image as starting points, and to model the two-dimensional state space equation from the upper left, upper right, lower right and lower left directions to build a two-dimensional state space Mamba model; wherein the modeling process of the two-dimensional state space equation is converted into the modeling of two independent one-dimensional state space equations;

[0043] A task execution module is used to use the two-dimensional state space Mamba model to perform recognition and prediction tasks in the field of computer vision; wherein the recognition and prediction tasks include image classification, target detection, semantic segmentation and instance segmentation.

[0044] The visual feature representation learning method and system based on the two-dimensional state space model of the embodiment of the present invention solves the problem that when the existing Mamba model is applied to the visual field, the simple image flattening operation often damages the modeling of local image features due to the local invariance of the image and the temporal characteristics of the Mamba model. The present invention uses a two-dimensional state space model to adaptively construct image feature representation, integrates the advantages of the Mamba model into visual processing, and retains the invariance of local features.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 is a flow chart of a visual feature representation learning method based on a two-dimensional state space model according to an embodiment of the present invention;

[0048] Figure 2 is an architecture diagram of a visual feature representation learning method based on a two-dimensional state space model according to an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of comparison between the present invention according to an embodiment of the present invention and other metric learning methods;

[0050] Figure 4 4 is a structural diagram of a visual feature representation learning system based on a two-dimensional state space model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0053] The following describes a visual feature representation learning method and system based on a two-dimensional state space model according to an embodiment of the present invention with reference to the accompanying drawings.

[0054] Figure 1 is a flow chart of a visual feature representation learning method based on a two-dimensional state space model according to an embodiment of the present invention. Figure 1 As shown, including:

[0055] S1, obtain input image;

[0056] S2, taking the four vertices of the input image as starting points, modeling the two-dimensional state space equation from the upper left, upper right, lower right and lower left directions to construct a two-dimensional state space Mamba model; wherein the modeling process of the two-dimensional state space equation is converted into the modeling of two independent one-dimensional state space equations;

[0057] S3, using the two-dimensional state space Mamba model to perform recognition prediction tasks in the field of computer vision; wherein the recognition prediction tasks include image classification, target detection, semantic segmentation, and instance segmentation.

[0058] Figure 2 It is the architecture diagram of the present invention, such as Figure 2 As shown, the present invention uses 2D SSM to perform feature encoding starting from four directions (upper left, upper right, lower right, and lower left).

[0059] It can be understood that the present invention obtains an input image to perform subsequent feature processing.

[0060] In one embodiment of the present invention, the basis of the one-dimensional state space equation Mamba model, using a hidden state vector to map one-dimensional inputs to their corresponding output results, is as follows:

[0061] h′(t)=Ah(t)+Bx(t),

[0062] y(t)=Ch(t),

[0063] Where x(t) and y(t) represent the input and output vectors respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices.

[0064] The above is a state space equation in continuous form. However, the input of most application scenarios in the field of artificial intelligence is in discrete form. The formula for discretization using zero-order hold technology is as follows:

[0065]

[0066] Where Δ is the time scale parameter, and are the discretized forms of A and B respectively. The one-dimensional state space equation in discrete form is shown below:

[0067]

[0068] yt=Ch t .

[0069] Among them, x t and t Represent the input and output vectors in discrete form, h t Represents the hidden state vector in discrete form.

[0070] In one embodiment of the present invention, for image samples, extending the state space model to a two-dimensional form constitutes a more essential feature extraction method, as shown below:

[0071]

[0072] Among them, h i，j 1 and h i，j 2 Represent the hidden state vectors in the horizontal and vertical directions respectively, and the meanings of other parameters are consistent with the one-dimensional state space equation. In the specific implementation, the present invention takes the modeling process in the horizontal direction as an example to illustrate, which is as follows:

[0073]

[0074] Expand the above formula into the following two parts:

[0075]

[0076] The first part is consistent with the form of the one-dimensional state space equation. The result of the first part is recorded as x i,j ′ and substitute it into the formula in the latter part:

[0077]

[0078] In order to simplify the calculation process, the present invention uses h i，j 2 The calculation process of is only related to the hidden state vector in the vertical direction, as follows:

[0079]

[0080] The above form is also related to the one-dimensional state space equation. Therefore, the present invention transforms the modeling process of the two-dimensional state space equation into the modeling of two independent one-dimensional state space equations, the difference being the setting of parameters (the former is A1, B1, the latter is A2, I).

[0081] The embodiment of the present invention adopts two-dimensional state space equations to model image features, thereby establishing a more reasonable and clever application of Mamba in the field of computer vision.

[0082] Furthermore, considering the non-sequential nature of image data, the present invention uses the four corners of the image as the starting positions to start the calculation of the two-dimensional state space equation. In implementation, the present invention rotates the input at different angles:

[0083] x i =rot (π / 2)·i (x), i = 0.1, 2, 3,

[0084] Then, the present invention splices the inputs rotated at different angles, as shown below:

[0085] z=concat([x0, x1, x2, x3], dim=0).

[0086] The spliced ​​results can be used to model the two-dimensional state space equation, and the modeled features will be aggregated according to their original positions before rotation.

[0087] Preferably, the present invention adopts an iterative method to model the two-dimensional state space equation, and the number of iterations is set to 24.

[0088] The embodiment of the present invention constructs a two-dimensional visual Mamba module, takes the four vertices of the image as starting points, and respectively performs modeling and calculation of the two-dimensional state space equations, thereby further enhancing the ability to understand the image.

[0089] Furthermore, after the model is built, the constructed two-dimensional Mamba model can be used to perform upstream recognition tasks and downstream dense prediction tasks in the field of computer vision, including image classification, object detection, semantic segmentation, and instance segmentation, so as to comprehensively measure the effect of this method.

[0090] Further, Figure 3 Schematic diagram of the comparison between the present invention and other metric learning methods. Figure 3 (a) shows that Mamba encodes text information using blocks based on a one-dimensional state space model after word segmentation. Figure 3 (b) in the figure shows that Visual Mamba (Vim) adopts a method of directly flattening the input image and then processing it through a one-dimensional state space model block similar to Mamba. Figure 3 (c) in the figure shows that the V2M framework proposed in the present invention introduces a two-dimensional state space model in the coding block without performing a flattening operation, which is more suitable for processing images.

[0091] The present invention uses two-dimensional state space equations to model images, thereby retaining the local invariance priors inherent in the image. Considering the non-temporal nature of the input image, the present invention simultaneously models the two-dimensional state space equations in four directions to enhance the representation capability of the model. The present invention evaluates the effectiveness of the framework in classification and various downstream tasks such as target detection, instance segmentation, and semantic segmentation, and the results prove the superiority of the two-dimensional visual Mamba model framework in the present invention in classification and migration performance.

[0092] The visual feature representation learning method based on the two-dimensional state space model according to the embodiment of the present invention solves the problem that when the existing Mamba model is applied to the visual field, the simple image flattening operation often damages the modeling of local image features due to the local invariance of the image and the temporal characteristics of the Mamba model. That is, the image feature representation is adaptively constructed using the two-dimensional state space model, the advantages of the Mamba model are integrated into the visual processing, and the local feature invariance is retained.

[0093] In order to implement the above embodiment, Figure 4 As shown, this embodiment also provides a visual feature representation learning 10 based on a two-dimensional state space model, including:

[0094] An image acquisition module 100 is used to acquire an input image;

[0095] The spatial model building module 200 is used to take the four vertices of the input image as starting points, and to model the two-dimensional state space equation from the upper left, upper right, lower right and lower left directions to build a two-dimensional state space Mamba model; wherein the modeling process of the two-dimensional state space equation is converted into the modeling of two independent one-dimensional state space equations;

[0096] The task execution module 300 is used to use the two-dimensional state space Mamba model to perform recognition prediction tasks in the field of computer vision; wherein the recognition prediction tasks include image classification, target detection, semantic segmentation and instance segmentation.

[0097] Exemplarily, the basis of the one-dimensional state space equation uses a hidden state vector to map a one-dimensional input to a corresponding output result as shown below:

[0098] h′(t)=Ah(t)+Bx(t),

[0099] y(t)=Ch(t),

[0100] Where x(t) and y(t) represent the input and output vectors respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices;

[0101] The formula for discretization using the zero-order hold technique is as follows:

[0102]

[0103] Where Δ is the time scale parameter, and are the discretized forms of A and B respectively. The one-dimensional state space equation form of the discrete form is as follows:

[0104]

[0105] yt=Ch t .

[0106] Among them, x t and t Represent the input and output vectors in discrete form, h t Represents the hidden state vector in discrete form.

[0107] Exemplarily, for the input image, the one-dimensional state space model is expanded to a two-dimensional form as follows:

[0108]

[0109] Among them, h i，j 1 and h i，j 2 Represent the hidden state vectors in the horizontal and vertical directions respectively. The modeling process in the horizontal direction is as follows:

[0110]

[0111] Expanded into the following two parts:

[0112]

[0113] The first part is consistent with the form of the one-dimensional state space equation. The result of the first part is recorded as x i,j ′ and substitute it into the formula in the latter part:

[0114]

[0115] h i，j 2 The calculation process of is only related to the hidden state vector in the vertical direction, as follows:

[0116]

[0117] For example, the input images in four directions are rotated at different angles:

[0118] x i =rot (π / 2)·i (x), i = 0, 1, 2, 3,

[0119] The input images rotated at different angles are stitched together as shown below:

[0120] z=concat([x0, x1, x2, x3], dim=0).

[0121] Exemplarily, the system is further used for:

[0122] Get the stitching results of the input images after being rotated at different angles;

[0123] Modeling the splicing result with a two-dimensional state space equation;

[0124] The modeled image features are aggregated according to their original positions before image rotation.

[0125] The visual feature representation learning system based on the two-dimensional state space model according to the embodiment of the present invention solves the problem that when the existing Mamba model is applied to the visual field, the simple image flattening operation often damages the modeling of local image features due to the local invariance of the image and the temporal characteristics of the Mamba model. That is, the image feature representation is adaptively constructed using the two-dimensional state space model, the advantages of the Mamba model are integrated into the visual processing, and the local feature invariance is retained.

[0126] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0127] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A visual feature representation learning method based on a two-dimensional state space model, It is characterized in that include: Get the input image; Taking the four vertices of the input image as starting points, modeling of the two-dimensional state space equation is performed from the upper left, upper right, lower right and lower left directions to construct a two-dimensional state space Mamba model; wherein the modeling process of the two-dimensional state space equation is converted into the modeling of two independent one-dimensional state space equations; The two-dimensional state space Mamba model is used to perform recognition prediction tasks in the field of computer vision; wherein the recognition prediction tasks include image classification, target detection, semantic segmentation and instance segmentation.

2. The method according to claim 1, It is characterized in that The basis of the one-dimensional state space equation uses a hidden state vector to map a one-dimensional input to a corresponding output result as shown below: h′(t)=Ah(t)+Bx(t), y(t)=Ch(t), Where x(t) and y(t) represent the input and output vectors respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices; The formula for discretization using the zero-order hold technique is as follows: Where Δ is the time scale parameter, and are the discretized forms of A and B respectively. The one-dimensional state space equation form of the discrete form is as follows: the t =Ch t· Among them, x t and t Represent the input and output vectors in discrete form, h t Represents the hidden state vector in discrete form.

3. The method according to claim 2, It is characterized in that For the input image, the one-dimensional state space model is expanded to a two-dimensional form as follows: Among them, h i,j 1 and h i,j 2 Represent the hidden state vectors in the horizontal and vertical directions respectively. The modeling process in the horizontal direction is as follows: Expanded into the following two parts: The first part is consistent with the form of the one-dimensional state space equation. The result of the first part is recorded as x i,j ′ and substitute it into the formula in the latter part: h i,j 2 The calculation process of is only related to the hidden state vector in the vertical direction, as follows:

4. The method according to claim 3, It is characterized in that Rotate the input images in four directions at different angles: x i =rot (π / 2)·i (x),i=0,1,2,3, The input images rotated at different angles are stitched together as shown below: z=concat([x 0 ,x 1 ,x 2 ,x 3 ],dim=0)。 5. The method according to claim 4, It is characterized in that The method further comprises: Get the stitching results of the input images after being rotated at different angles; Modeling the splicing result with a two-dimensional state space equation; The modeled image features are aggregated according to their original positions before image rotation.

6. A visual feature representation learning system based on a two-dimensional state space model, It is characterized in that include: An image acquisition module, used for acquiring an input image; A spatial model building module is used to take the four vertices of the input image as starting points, and to model the two-dimensional state space equation from the upper left, upper right, lower right and lower left directions to build a two-dimensional state space Mamba model; wherein the modeling process of the two-dimensional state space equation is converted into the modeling of two independent one-dimensional state space equations; A task execution module is used to use the two-dimensional state space Mamba model to perform recognition and prediction tasks in the field of computer vision; wherein the recognition and prediction tasks include image classification, target detection, semantic segmentation and instance segmentation.

7. The system according to claim 6, It is characterized in that The basis of the one-dimensional state space equation uses a hidden state vector to map a one-dimensional input to a corresponding output result as shown below: h′(t)=Ah(t)+Br(t), y(t)=Ch(t), Where x(t) and y(t) represent the input and output vectors respectively, h(t) represents the hidden state vector, A represents the evolution parameter matrix of the one-dimensional state space equation, and B and C represent the projection parameter matrices; The formula for discretization using the zero-order hold technique is as follows: Where Δ is the time scale parameter, and are the discretized forms of A and B respectively. The one-dimensional state space equation form of the discrete form is as follows: the t =Ch t· Among them, x t and t Represent the input and output vectors in discrete form, h t Represents the hidden state vector in discrete form.

8. The system according to claim 7, It is characterized in that For the input image, the one-dimensional state space model is expanded to a two-dimensional form as follows: Among them, h i,j 1 and h i,j 2 Represent the hidden state vectors in the horizontal and vertical directions respectively. The modeling process in the horizontal direction is as follows: Expanded into the following two parts: The first part is consistent with the form of the one-dimensional state space equation. The result of the first part is recorded as x i,j ′ and substitute it into the formula in the latter part: h i,j 2 The calculation process of is only related to the hidden state vector in the vertical direction, as follows:

9. The system according to claim 8, It is characterized in that Rotate the input images in four directions at different angles: x i =rot (π / 2)·i (x),i=0,1,2,3, The input images rotated at different angles are stitched together as shown below: z=concat([x 0 ,x 1 ,x 2 ,x 3 ],dim=0)。 10. The system according to claim 9, It is characterized in that The system is also used for: Get the stitching results of the input images after being rotated at different angles; Modeling the splicing result with a two-dimensional state space equation; The modeled image features are aggregated according to their original positions before image rotation.

Images