Lightweight graph matching model training method based on state space model, graph matching method and system

Abstract

The invention provides a lightweight graph matching model training method and system based on a state space model, and a graph matching method and system, and belongs to the technical field of computer vision. And processing the obtained image pair by using a pre-trained matching model to obtain an image matching result. According to the invention, occupation of a model reasoning video memory is reduced, and only one read-write operation is needed; the parameter quantity of the model is extremely low, less hardware resources are occupied, and deployment and reasoning in an environment where the hardware resources are limited are facilitated; the graph task is subjected to dimension reduction to form the sequence prediction task, and feature learning is performed, so that the calculation amount required during model training and reasoning is effectively reduced, the video memory consumption required during reasoning is also reduced, and meanwhile, the matching precision is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a lightweight graph matching model training method based on a state space model, a graph matching method and a system. BACKGROUND

[0002] As a key technology for graph data processing, graph matching promotes the development of graph representation learning, provides a new idea for solving non-Euclidean space data problems, and promotes the perfection of the geometric deep learning theory system. In practical applications, graph matching technology has been widely used in computer vision, bioinformatics, social network analysis and other fields, and important scenarios such as medical image analysis, protein network alignment and cross-platform user identification. Graph matching can not only effectively handle the irregularity of graph structure data, but also has great value in many industrial applications.

[0003] Traditional graph matching methods mainly fall into two categories: (1) combinatorial optimization methods: model the matching problem as a quadratic assignment problem and solve it using integer programming, spectral methods, etc. This kind of method has strong theoretical guarantee, but the computational complexity is as high as O(n3) or higher, which makes it difficult to extend to large-scale graphs; (2) deep learning-based methods: use deep learning and graph neural networks to automatically learn graph features and matching strategies. Although the performance is superior, the graph neural networks of existing methods are over-parameterized, with parameter quantities often reaching millions, which cannot meet the real-time requirements of mobile / edge devices.

[0004] Current lightweight techniques mainly focus on deep neural network model compression, including pruning, quantization, knowledge distillation, etc. However, direct application of these techniques to graph matching will result in a significant decline in topological awareness, for example, pruning may destroy the key adjacency matrix structure, and quantization may lose the subtle differences between node features.

[0005] Although the deep learning-based graph neural network (GNN) method performs well in graph structure data modeling, its actual deployment still faces significant challenges, especially in resource-constrained edge computing scenarios. The single-layer parameter quantity of typical GNN architectures (such as GAT and GIN) can reach millions, and the model size grows exponentially after multi-layer stacking. For example, the parameter quantity of the standard GraphSAGE exceeds 4.7M when processing the Cora dataset in the full connection configuration, and the parameter quantity of the more complex GATv2 model on the OGB-arxiv dataset is as high as 21M. The parameter quantity directly leads to the expansion of the model file size, which exceeds 500MB, far exceeding the carrying capacity of the available memory of the edge device Raspberry Pi 4B, which is usually only 1-4GB. These limitations make it difficult for existing GNN methods to be deployed in typical edge scenarios such as smart cameras and wearable devices.(2) Direct application of network lightweight techniques (such as pruning and quantization) to the model can cause loss of key topological feature information, severely affecting the performance of graph matching. Specifically, although these lightweight methods can effectively compress the model size and improve the inference speed, they can destroy the topological relationships and high-order semantic features in the graph structure data. Pruning operations can remove neuron connections sensitive to topological features; the quantization process can introduce numerical errors, causing deviations in the similarity calculation between nodes, ultimately leading to a decrease in the accuracy of graph matching. SUMMARY

[0006] The present application aims to provide a lightweight graph matching method and system based on a state space model to solve at least one of the technical problems in the background art.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a lightweight graph matching model training method based on a state space model, comprising:

[0009] Obtain multiple open-source datasets of graph matching tasks as a training set, preprocess the training set: select photo pairs A and B with the same number of point pairs in the same category of pictures, record the true matching relationship of the key point sequence in the photo pair, and then shuffle the two key point feature sequences;

[0010] Convert the point pair feature sequence obtained after preprocessing into a tensor data type that can be transferred and learned by a neural network, randomly set zero to the initialized key point feature sequence and position feature sequence of graph A and graph B, and perform high-dimensional mapping, element sum the key point features and position features to form an initial feature sequence that can be updated and learned;

[0011] The initial feature sequence of figure A and figure B is respectively expanded in row and column to form four different expanded sequences, the adjacency matrix structure of the figure is learned, and the initial feature sequence of the two figures is updated by using the adjacency matrix;

[0012] The updated feature sequence of figure A and figure B is spliced to form a fused feature sequence for mutual learning, the prediction path of the sequence is reconstructed, the matching matrix is learned, different matching matrices are output for each path, the element level sum of all output matching matrices is calculated, the average is calculated and the final soft corresponding matching matrix is calculated, the soft corresponding matching matrix is used to update the feature sequence of figure A and figure B again.

[0013] The predicted matching matrix is calculated by the Hungarian algorithm to obtain a discrete Boolean matrix representing the one-to-one correspondence of the key points of figure A and figure B.

[0014] The obtained Boolean matrix is used to calculate the loss in the model learning process by using the cross entropy loss function, and the gradient is back propagated to train and update the model weight and optimize in a loop.

[0015] As a further limitation of the first aspect of the application, the VGG16 network is used to extract the key point visual features, and the key point visual features and position information are encoded and randomly zeroed in the feature dimension; the key point coordinates are converted into high-dimensional position embedding, the position encoding is 2-dimensional mapping to 128-dimensional, the input two key point feature sequences and position feature sequences are converted into tensor type, and the key point feature sequence and position feature sequence are element level summed to obtain the enhanced feature sequence.

[0016] As a further limitation of the first aspect of the application, the initial nxd-dimensional initial feature sequence is expanded, and the expansion is performed in the row and column directions respectively with a point sequence length n times to obtain n 2 ×d-dimensional feature sequence, for a key point sequence of a figure, two different n 2 ×d-dimensional feature sequences are generated, each picture pair contains four n 2 ×d-dimensional feature sequences of two figures; the expanded four feature sequences are learned for the graph structure feature, i.e. the adjacency matrix, to obtain four n×n adjacency matrices predicted by row and column respectively, and the adjacency matrices of figure A and figure B predicted by row and column respectively, a total of four n×n adjacency matrices predicted by row and column respectively; for each figure pair corresponding to the different row and column expansion, there are two different adjacency matrices corresponding thereto, the element sum of the two adjacency matrices is calculated and averaged to form a symmetric matrix, i.e. the final adjacency matrix; the operation is performed on figure A and figure B at the same time, and the initial feature sequence corresponding to each is updated by using the respective adjacency matrix, and the element sum of the position feature sequence is calculated to obtain the updated feature sequence of figure A and figure B.

[0017] As a further limitation of the first aspect of the application, the two updated feature sequences are expanded in row and column dimensions, and the expanded feature sequences are spliced in the feature dimension to obtain an n 2 ×2d-dimensional fusion feature sequence; two zigzag continuous prediction paths are artificially set, and the fusion feature sequence is rearranged according to the prediction paths and sent into a multi-path Mamba module for matching matrix prediction, each path outputting a prediction result of a matching matrix; each output matching matrix is averaged and subjected to a Sinkhorn algorithm to obtain a final n×n soft correspondence relationship matrix, i.e., a model output; the matching matrix is used to update the two feature sequences of graph A and graph B, respectively, and finally the updated feature sequences are summed with the updated feature sequences of the adjacency matrix and sent into a RELU function for activation as the initial feature sequences for the next round.

[0018] As a further limitation of the first aspect of the application, the predicted soft matching matrix and the real matching matrix are vectorized, a cross-entropy loss function is used to calculate the model loss, a weight w=5 is used to balance the positive and negative samples, the loss is used to quantify the difference between the prediction result of the model and the real result, the optimal value of the model parameters is found by minimizing the loss function, thereby improving the prediction accuracy of the model; during the training process, the weighted cross-entropy is calculated as a supervision signal, the model parameters are optimized through gradient backpropagation, and the model parameters are continuously adjusted to reduce the value of the loss function, and the model gradually learns the inherent rules and characteristics of the graph feature data.

[0019] In a second aspect, the application provides a lightweight graph matching method, comprising:

[0020] obtaining a pair of images to be matched;

[0021] processing the obtained pair of images using a pre-trained matching model to obtain a graph matching result; wherein the matching model is obtained using the model training method of the first aspect.

[0022] In a third aspect, the application provides a lightweight graph matching system, comprising:

[0023] an obtaining module for obtaining a pair of images to be matched;

[0024] a processing module for processing the obtained pair of images using a pre-trained matching model to obtain a graph matching result; wherein the matching model is obtained using the model training method of the first aspect.

[0025] In a fourth aspect, the present application provides a non-transitory computer readable storage medium for storing computer instructions, which, when executed by a processor, implement the lightweight graph matching method according to the second aspect.

[0026] In a fifth aspect, the present application provides a computer device comprising a memory and a processor, the processor and the memory being in communication with each other, the memory storing program instructions executable by the processor, and the processor invoking the program instructions to execute the lightweight graph matching method according to the second aspect.

[0027] In a sixth aspect, the present application provides an electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute instructions for implementing the lightweight graph matching method according to the second aspect.

[0028] Graph matching: Graph matching is an important task in computer vision, pattern recognition, and machine learning, aiming to find the node correspondence between two graph structures so that their topological structures and node / edge attributes are as consistent as possible.

[0029] State space model (SSM): SSM is a mathematical model used to describe dynamic systems, widely applied in control theory, signal processing, time series analysis, machine learning, etc. It links the state (a set of hidden variables) of the system with the observed data, describing the evolution process of the system through state equations and observation equations.

[0030] Lightweight: Lightweight refers to the implementation of efficient, low-power, and low-latency computing tasks in resource-constrained environments such as mobile devices, embedded systems, edge computing, etc. through algorithm optimization, model compression, or hardware adaptation. Its core goal is to balance performance and resource consumption, suitable for scenarios with high real-time requirements or harsh deployment conditions.

[0031] Graph neural network: Graph neural network is a class of deep learning models specifically designed for processing graph-structured data, capable of capturing the relationships between nodes (edges) and global topological information, widely applied in social networks, molecular structures, recommendation systems, etc.

[0032] The application has the advantages of reducing model inference memory occupation, completing only once read-write operation, extremely low model parameter quantity, less hardware resource occupation, being more conducive to deployment and inference in hardware resource limited environment, reducing calculation quantity required during model training and inference, reducing memory consumption required during inference, and ensuring matching accuracy.

[0033] The advantages of the additional aspects of the application will be more apparent from the following description section or be understood through the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0035] Figure 1 The flow chart of the light-weighted graph matching method based on the state space model is described in the embodiments of the application.

[0036] Figure 2 The schematic diagram of the direction Mamba module network structure is described in the embodiments of the application. DETAILED DESCRIPTION

[0037] The embodiments of the application will be described in detail below, and the examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with the drawings are exemplary and are only used to explain the application, and cannot be explained as the limitation of the application.

[0038] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art in the field of the application.

[0039] It should also be understood that terms such as those defined in general dictionaries should be understood as having meanings consistent with those in the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as such.

[0040] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprising," "including," "containing," or "having" and the like, when used in the specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0041] In the description of the present application, the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like, mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the present application. Such phrases do not necessarily refer to the same embodiment or example, and such phrases do not necessarily refer to any one or the same embodiment or example. Further, the description can use the term "embodiment" or "example" to refer to a specific embodiment or example. This does not necessarily mean that a particular feature, structure, material, or characteristic is required. In addition, the description can refer to a "plurality" of a feature, structure, material, or characteristic. This does not necessarily mean that there is more than one of the feature, structure, material, or characteristic. In other words, a "plurality" of a feature, structure, material, or characteristic can mean that there are one or more of the feature, structure, material, or characteristic.

[0042] In order to understand the present application, the following further explains the present application with specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present application.

[0043] Those skilled in the art should understand that the drawings are only schematic views of the embodiments, and the components in the drawings are not necessarily essential for implementing the present application.

[0044] The present application aims at the application bottleneck of existing graph matching methods in resource-constrained scenarios, and proposes a lightweight graph matching network design method based on a state space model (SSM). Traditional graph matching networks usually rely on complex network architecture and are difficult to run in real time on lightweight devices. Existing deep learning models have large parameters and high inference memory occupation. Secondly, some lightweight methods such as pruning and quantization applied to graph matching will significantly reduce the accuracy, especially for topological structure sensitive tasks, due to the lack of efficient feature transmission mechanism, resulting in serious loss of structure information. In order to solve these problems, the present application proposes a design method of graph neural network, which creatively introduces the time series modeling capability of SSM into the optimization of the graph matching task through the modeling method of serialization, reduces the two-dimensional graph matching task to one-dimensional sequence prediction task, and uses the time series modeling capability of SSM to process the graph structure information. At the same time, the present application greatly reduces the required parameter quantity and memory consumption during inference, and maintains the matching accuracy compared with existing methods; reduces the resource consumption of the graph matching task, and expands the application range of the graph matching task, so that it can better meet the needs in actual scenarios. The present application uses SSM based on Mamba to design the network structure. Due to the architecture of SSM, the present application greatly reduces the parameter quantity of the model; avoids O(n2) calculation consumption; maintains the inference speed and accuracy; and the hardware-friendly perception of Mamba significantly reduces the memory consumption during inference, and can run in some resource-constrained environments.

[0045] Embodiment 1

[0046] In this embodiment 1, a lightweight graph matching system is first provided, comprising: an acquisition module for acquiring a pair of images to be matched; a processing module for processing the acquired pair of images using a pre-trained matching model to obtain a graph matching result.

[0047] In this embodiment, a lightweight graph matching method is implemented using the above system, comprising: acquiring a pair of images to be matched using the acquisition module; and processing the acquired pair of images using a pre-trained matching model using the processing module to obtain a graph matching result.

[0048] As shown in Figure 1 In this embodiment, the matching model is a lightweight graph matching model based on a state space model, and the training method thereof comprises the following steps:

[0049] S1: Use three open source datasets of graph matching tasks, Willow, Pascal VOC, Spair-71K, as training and test samples. The preprocessing of model data samples mainly includes: screening the same number of point pairs in the pictures of the same class: picture A and picture B; recording the true matching relationship of the key point sequence in the picture pair, and then shuffling the two key point feature sequences and sending them into the model for training. In the test, a certain number of samples of each category are selected from the test samples for testing, and the prediction accuracy of the model is calculated.

[0050] S2: The model receives the initialized point pair feature sequence input, converts it into a tensor data type that can be transmitted and learned by the neural network, and randomly sets zero to the initialized key point feature sequence and position feature sequence of picture A and picture B, respectively, and performs high-dimensional mapping. The key point features and position features are summed to form an initial feature sequence that can be updated and learned.

[0051] S3: The initial feature sequences of picture A and picture B are respectively expanded in rows and columns to form four different expanded sequences, which are sent into four direction Mamba modules (deep learning modules based on state space model (SSM)) to learn the adjacency matrix structure of the graph, and the initial feature sequences of the two graphs are updated using the adjacency matrix.

[0052] S4: The updated feature sequences of picture A and picture B are spliced to form a fused feature sequence that learns from each other, and two different continuous zigzag prediction trajectories are artificially set to reconstruct the prediction path of the sequence, which is sent into the multi-path Mamba module for matching matrix learning. Different matching matrices are output for each path, and finally all the output matching matrices are summed at the element level, averaged, and the final soft correspondence matrix is obtained through the Sinkhorn algorithm. Finally, the soft correspondence matching matrix obtained is used to update the feature sequences of picture A and picture B again.

[0053] S5: The matching matrix predicted by the model is calculated through the Hungarian algorithm to obtain a discrete Boolean matrix representing the one-to-one correspondence of the key points of picture A and picture B.

[0054] S6: The obtained Boolean matrix is used to calculate the Loss in the model learning process through the cross-entropy loss function, and the gradient is backpropagated to train and update the model weights, which are optimized in a loop to continuously improve the prediction accuracy of the model.

[0055] S7: The obtained Boolean matrix is compared with the true point pair relationship recorded during preprocessing of the sample to calculate the matching accuracy of the model prediction, which is compared with other graph matching to verify the accuracy of the model prediction.

[0056] The step S1 specifically comprises the following steps:

[0057] S11: Split the entire dataset into training set and test set according to the ratio of 8:2. The training set is used for model training; the test set is used for model performance evaluation. When training, a class is randomly selected from the training set, and two pictures with the same number of key points are randomly selected from the class to form a pair of pictures to be predicted: picture A and picture B.

[0058] S12: In order to avoid the model memorizing specific sequences during training, the recorded key point correspondence needs to be shuffled. This operation can increase the randomness of training and make the model more robust.

[0059] S13: When testing, 1000 picture pairs are uniformly selected from each class in the test set for model prediction and prediction accuracy calculation, and finally the average accuracy of all classes is calculated to obtain the test accuracy of the model on the dataset.

[0060] The step S2 specifically comprises the following steps:

[0061] S21: Use VGG16 network to extract key point visual features, and encode key point visual features and position information for random zero in feature dimension, with a ratio of 0.3 to reduce overfitting. The designed learnable position encoder is a fully connected layer network structure, which is used to upgrade the two-dimensional coordinates of the image to the feature dimension matching the visual features. After upgrading, the key point feature sequence and the position feature sequence are summed at the element level to obtain an enhanced feature sequence.

[0062] In each training process, iterative optimization is performed, and the specific formula is as follows:

[0063] S A =MLP(F A )+MLP(P A ),

[0064] Wherein, S A ∈R n×d , n represents the number of key points, d represents the feature dimension, and the position encoder is a multi-layer perception (MLP) used to map two-dimensional coordinates to d-dimensional embedding vectors. The same MLP is applied to picture B to generate a feature sequence S B ∈R n×d .

[0065] S22: input the enhanced feature sequence into the designed network for training, the batchsize of training is equal to 32, the layer of model is equal to 3, the internal feature dimension of the full connection layer of model is 128, the feature dimension of the used Mamba module is 1024, the training round is set to 200 rounds, each round iteration is 100 times, the initial training learning rate of model is 0.001, and the learning rate is continuously decayed by 0.98 after each round of training until the fixed value 0.0005 is no longer decreased.

[0066] The step S3 specifically comprises the following steps:

[0067] S31: expand the initial nxd-dimensional initial feature sequence, expand by n times in the row and column directions respectively to obtain n 2 xd-dimensional feature sequence, for a key point sequence of a picture, two different n 2 xd-dimensional feature sequences are generated, and each picture contains four n 2 xd-dimensional feature sequences.

[0068] S32: then, the expanded four feature sequences are respectively sent into the designed direction Mamba module, as shown in the figure, Figure 2 The module receives an input sequence (B, N, dim=d) and generates an output sequence (B, N, dim=1). The specific structure is that the input sequence first passes through two linear projection layers, one branch of which enters the local convolution layer to capture local features and enhance its understanding of sequence data, and then connects the SSM layer. The output is combined with a branch, a nonlinear activation function is applied to increase the expression ability of the model, and finally enters the linear projection layer to obtain the output. The learning (predicting the probability of the existence of edges between the points themselves) of the graph structure feature, i.e. the adjacency matrix, is performed. The output graph A is the adjacency matrix predicted by row and the adjacency matrix predicted by column, and the adjacency matrix predicted by row and the adjacency matrix predicted by column of graph B, a total of four n×n adjacency matrices predicted by row and column.

[0069] S33: for each picture corresponding to the row and column expansion, there are two different adjacency matrices corresponding thereto, the element sum of the two adjacency matrices is calculated and averaged to form a symmetric matrix, i.e. the final adjacency matrix. The operation is performed on graph A and graph B at the same time, and the respective adjacency matrices are used to update the corresponding initial feature sequences, and the element sum of the position feature sequence is obtained to obtain the updated feature sequences of graph A and graph B.

[0070] The step S4 specifically comprises the following steps:

[0071] S41: expand the row and column dimensions of the updated two feature sequences, splice the expanded feature sequences in the feature dimension to obtain n2 x2d-dimensional fused feature sequence.

[0072] S42: two zigzag continuous prediction paths are artificially set, and the fused feature sequence is rearranged according to the prediction paths and sent to the multi-path Mamba module for matching matrix prediction, and each path outputs a prediction result of a matching matrix.

[0073] S42: the output of each matching matrix is averaged and subjected to Sinkhorn algorithm (Sinkhorn algorithm parameter tau is 0.1) to obtain a final n x n soft correspondence matrix, i.e. model output, and the matching matrix is used to update the two feature sequences of graph A and graph B respectively, and finally the updated feature sequences are summed with the updated feature sequences of the adjacency matrix and sent to the RELU function for activation as the initial feature sequence of the next round.

[0074] The step S5 specifically includes the following steps: the predicted soft matching matrix is subjected to Hungarian algorithm to obtain a discrete matching result, forming a Boolean matrix, i.e. the final matching matrix, which is used for subsequent loss function calculation and accuracy verification.

[0075] The step S6 specifically includes the following steps: the predicted soft matching matrix and the real matching matrix are vectorized, and a cross-entropy loss function is used to calculate the model loss, using a weight w = 5 to balance positive and negative samples. This loss is used to quantify the difference between the predicted result and the real result of the model, and by minimizing the loss function, the optimal value of the model parameter is found, thereby improving the prediction accuracy of the model. During the training process, the weighted cross-entropy is calculated as a supervision signal, and the model parameters are optimized through gradient backpropagation, and the model parameters are adjusted to reduce the value of the loss function. The model gradually learns the internal rules and characteristics of the graph feature data, so as to be able to more accurately predict.

[0076] The step S7 specifically includes the following steps: the obtained Boolean matrix is compared with the real point correspondence recorded in the sample preprocessing stage, and the ratio of the number of correctly matched point pairs to the total number of point pairs is calculated as an accuracy index. In this way, not only the accuracy of the model can be quantified, but also the prediction results of other graph matching algorithms can be systematically compared. The memory usage and model parameter amount of different models during inference are recorded and compared, and the lightweight effect is verified by comparing with the baseline method, proving that the lightweight design of the model has achieved better results in actual inference, and can complete the same graph matching task with less resource consumption.

[0077] In summary, the advantages of the method described in this embodiment are described from the following two aspects. Graph neural network algorithm aspect: In the graph neural network, the model and operation are processed on the SRAM by utilizing the small memory occupation characteristic of the SSM model itself. Compared with the network structure designed based on the attention mechanism, the memory occupation of the model in this embodiment is greatly reduced during inference. Since the attention mechanism needs to perform a large number of reading operations, the high memory requirement caused by the attention matrix requires repeated reading and writing operations, while the model in this embodiment only needs one reading and writing operation. At the same time, the parameter amount of the model is extremely low, and the occupation of hardware resources is less, which is more conducive to deployment and inference in an environment with limited hardware resources. Graph matching task aspect: The graph task is creatively reduced to a sequence prediction task and feature learning, effectively reducing the amount of calculation required during model training and inference, and also reducing the memory consumption required during inference, while achieving an accuracy effect comparable to existing methods, greatly expanding the application prospect of the graph matching task.

[0078] Embodiment 2

[0079] Embodiment 2 provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement a lightweight graph matching method as described above. The method includes: obtaining a pair of images to be matched; processing the obtained pair of images using a pre-trained matching model to obtain a graph matching result; wherein the matching model is obtained using the model training method described in Embodiment 1.

[0080] Embodiment 3

[0081] Embodiment 3 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute a lightweight graph matching method based on a state space model as described above. The method includes: obtaining a pair of images to be matched; processing the obtained pair of images using a pre-trained matching model to obtain a graph matching result; wherein the matching model is obtained using the model training method described in Embodiment 1.

[0082] Embodiment 4

[0083] The embodiment 4 provides an electronic device, comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device runs, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing a state space model-based lightweight graph matching method as described above, the method comprising: acquiring an image pair to be matched; processing the acquired image pair by using a pre-trained matching model to obtain a graph matching result; wherein the matching model is obtained by using the model training method in the embodiment 1.

[0084] In summary, in view of the large memory consumption of the traditional network reasoning, the large model parameter quantity, and the inherent defects of the traditional lightweight technology, the present application fully utilizes the efficient sequence modeling capability and the structured parameter compression characteristics of the Mamba architecture, reduces the parameter quantity of the traditional graph matching model by more than 90% through the sub-linear complexity calculation characteristics of the Mamba architecture and the serialized graph structure representation method, and significantly reduces the storage and calculation overhead; meanwhile, the time sequence dependence modeling capability of the SSM is utilized to automatically capture the dynamic association rules between different object graph structures, and the feature information loss problem caused by the traditional pruning / quantization method is effectively avoided. The Mamba hardware-friendly perception mode greatly reduces the memory access frequency, so that the model can still maintain efficient reasoning on the mobile terminal and the edge device, and meet the real-time requirement. The present application shows excellent applicability in resource-constrained scenes such as mobile terminal applications (such as AR scene understanding and real-time visual positioning) and industrial Internet of Things (such as device topology matching). Compared with the traditional scheme, the present method reduces the memory and computing power consumption while maintaining high-precision graph matching capability, provides a reliable solution for graph matching data processing in the edge computing environment, and has important industrial application value.

[0085] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can be embodied in the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer usable program code embodied thereon.

[0086] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0087] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks. Figure 1 one or more flow or flows and / or block diagram block or blocks.

[0089] The above description is only a specific implementation of the present application, and is not intended to limit the protection scope of the present application. It should be understood by those skilled in the art that various modifications or changes can be made to the disclosed technical solutions without departing from the spirit and scope of the present application.

Claims

1. A lightweight graph matching model training method based on a state-space model, characterized in that, include: We obtained open-source datasets for multiple graph matching tasks as training sets and preprocessed the training sets: we selected photo pairs A and B with the same number of point pairs in the same category, recorded the true matching relationship of the key point sequences in the photo pairs, and then shuffled the feature sequences of the two key points. The preprocessed point-pair feature sequence is converted into a tensor data type that can be learned by the neural network. The initial key point feature sequence and position feature sequence of Figure A and Figure B are randomly set to zero respectively, and high-dimensional mapping is performed. The key point features and position features are summed element by element to form an initial feature sequence that can be updated and learned. The initial feature sequences of graphs A and B are expanded by rows and columns respectively to form four different expanded sequences. The adjacency matrix structure of the graphs is learned, and the initial feature sequences of the two graphs are updated using the adjacency matrix. The updated feature sequences of Figure A and Figure B are concatenated to form a fused feature sequence that learns from each other. The prediction path of the sequence is reconstructed, and the matching matrix is ​​learned. Different matching matrices are output for each path. All output matching matrices are summed element-wise, averaged, and the final soft correspondence matching matrix is ​​calculated. The feature sequences of Figure A and Figure B are updated again using the obtained soft correspondence matching matrix. The predicted matching matrix is ​​calculated using the Hungarian algorithm to obtain a discrete Boolean matrix that represents the one-to-one correspondence between the key points of Figure A and Figure B. Using the obtained Boolean matrix, the loss during the model learning process is calculated through the cross-entropy loss function, and gradient backpropagation is performed to train and update the model weights, and the process is iterated and optimized.

2. The lightweight graph matching model training method based on a state-space model according to claim 1, characterized in that, The VGG16 network is used to extract keypoint visual features, and the keypoint visual features and location information are encoded and randomly zeroed in the feature dimension. The keypoint coordinates are converted into high-dimensional location embeddings, and the location is encoded as a 2D mapping to 128D. The two input keypoint feature sequences and location feature sequences are converted into tensor types, and the keypoint feature sequences and location feature sequences are summed element-wise to obtain the enhanced feature sequence.

3. The lightweight graph matching model training method based on a state-space model according to claim 2, characterized in that, The initial n×d dimensional feature sequence is expanded by multiplying the length of the point sequence by n in both the row and column directions, resulting in an n2×d dimensional feature sequence. For a keypoint sequence of an image, two different n2×d dimensional feature sequences are generated, resulting in four n2×d dimensional feature sequences for each image pair. The expanded four feature sequences are then used to learn graph structure features, i.e., adjacency matrices, to obtain the adjacency matrices predicted by rows and columns for image A, and by rows and columns for image B, for a total of four n×n adjacency matrices predicted by rows and columns respectively. For each image with different row and column expansions, there are two different corresponding adjacency matrices. These two adjacency matrices are summed element-wise and averaged to form a symmetric matrix, i.e., the final adjacency matrix. This operation is performed simultaneously on images A and B, and the corresponding initial feature sequences are updated using their respective adjacency matrices. The updated feature sequences for images A and B are then summed element-wise with the positional feature sequences.

4. The lightweight graph matching model training method based on a state-space model according to claim 3, characterized in that, The updated two feature sequences are expanded in row and column dimensions, and then concatenated along these dimensions to obtain an n2×2d dimensional fusion feature sequence that can be used to learn matching relationships. Two zigzag continuous prediction paths are manually defined, and the fusion feature sequence is rearranged according to the prediction paths and fed into the multi-path Mamba module for matching matrix prediction. Each path outputs a prediction result for a matching matrix. The average of each output matching matrix is ​​then passed through the Sinkhorn algorithm to obtain the final n×n soft correspondence matrix, which is the model output. The matching matrix is ​​used to update the two feature sequences of Figure A and Figure B respectively. Finally, the updated feature sequence is summed element-wise with the feature sequence updated by the adjacency matrix and fed into the ReLU function for activation, serving as the initial feature sequence for the next round.

5. The lightweight graph matching model training method based on a state-space model according to claim 4, characterized in that, The predicted soft-match matrix and the true matching matrix are vectorized, and the model loss is calculated using the cross-entropy loss function. The positive and negative samples are balanced using a weight w=5. This loss is used to quantify the difference between the model's prediction and the true result. By minimizing the loss function, the optimal values ​​of the model parameters are found, thereby improving the model's prediction accuracy. During training, weighted cross-entropy is calculated as a supervision signal, and the model parameters are optimized through gradient backpropagation. The model parameters are continuously adjusted to reduce the value of the loss function, and the model gradually learns the inherent rules and features of the graph feature data.

6. A lightweight graph matching method, characterized in that, include: Get the image pairs to be matched; The acquired image pairs are processed using a pre-trained matching model to obtain graph matching results; wherein the matching model is obtained using the model training method described in any one of claims 1-5.

7. A lightweight graph matching system, characterized in that, include: The acquisition module is used to acquire image pairs to be matched; The processing module is used to process the acquired image pairs using a pre-trained matching model to obtain image matching results; wherein the matching model is obtained using the model training method described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the lightweight graph matching method as described in claim 6.

9. A computer device, characterized in that, It includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions that can be executed by the processor, and the processor calling the program instructions to execute the lightweight graph matching method as described in claim 6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions that implement the lightweight graph matching method as described in claim 6.

Images