Circuit breaker image matching method based on improved SuperGlue

Abstract

The present application discloses a circuit breaker image matching method based on an improved SuperGlue, which relates to the field of image matching technology, including: for any circuit breaker image to be matched, generating a feature vector of the circuit breaker image to be matched according to the feature point position and visual descriptor of the circuit breaker image to be matched; in each attention layer, fusing the input vector generated based on the feature vector, and transmitting the fused input vector to the next cross attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched; after traversing each circuit breaker image to be matched to obtain each feature representation, processing each feature representation through the optimal matching layer to obtain the matching relationship between each circuit breaker image to be matched. The present application performs multi-head attention fusion processing in the self-attention layer, which improves the accuracy and robustness of matching circuit breaker images using the SuperGlue algorithm.

Description

Technical Field

[0001] The present application relates to the field of image matching technology, and in particular to a circuit breaker image matching method, device, electronic device, storage medium and computer program product based on improved SuperGlue. Background Art

[0002] During the circuit inspection process, operators usually operate the circuit breaker pressure plate to cut off the current that needs to be cut off, and take pictures of the circuit breaker and upload them to the system. The management personnel will check the circuit breaker pictures uploaded to the system and determine whether the operator's operation is compliant.

[0003] The existing technical solutions mainly rely on manual inspection. However, the visual inspection of managers is affected by many factors. For example, there are many types of circuit breakers, and there are various types of protective pressure plates on the circuit breakers, which have different functions and complex logic of the throw-in and throw-out states, which makes it extremely difficult for managers to judge whether the pressure plates are operated correctly. In addition, due to the limitations of inspection conditions, the pictures of circuit breakers taken by operators in special environments, such as outdoor suspension, are subject to light pollution, scale changes, angle deviation, and incomplete pictures, which affect the judgment of managers. As the duration of manual inspection increases, managers are prone to fatigue and misjudgment. Therefore, there are also technical solutions that rely on image matching algorithms, such as the use of the open source SuperGlue algorithm, which imitates the process of matching two objects by the human eye through self-attention and cross-attention (when the human eye is looking for similar relationships between two objects, it will constantly compare the feature points between the two objects and the feature points of the objects themselves). However, in the actual use of the algorithm to train circuit breaker pictures and match the uploaded circuit breaker pictures with standard pictures, it is found that the algorithm has a low accuracy rate in matching complex circuit breaker pictures.

[0004] Therefore, how to improve the accuracy of matching circuit breaker images using the SuperGlue algorithm is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a circuit breaker image matching method, device, electronic device, storage medium and computer program product based on improved SuperGlue, aiming to solve the technical problem of how to improve the accuracy of matching circuit breaker images using the SuperGlue algorithm.

[0006] To achieve the above objectives, the present application proposes a circuit breaker image matching method based on an improved SuperGlue, wherein the circuit breaker image matching method based on the improved SuperGlue is applied to a SuperGlue feature matching network, wherein the SuperGlue feature matching network includes an attention graph neural network and an optimal matching layer, wherein the attention graph neural network includes at least a plurality of alternating self-attention layers and cross-attention layers, and the circuit breaker image matching method based on the improved SuperGlue includes:

[0007] For any one of the to-be-matched circuit breaker images, obtain a feature point position and a visual descriptor of the to-be-matched circuit breaker image, and generate a feature vector of the to-be-matched circuit breaker image according to the feature point position and the visual descriptor;

[0008] In each attention layer, the input vectors generated based on the feature vectors are fused, and the fused input vectors are transmitted to the next cross attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched;

[0009] After traversing each circuit breaker image to be matched to obtain each feature representation, the feature representation is processed by the optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched.

[0010] In one embodiment, the step of fusing the input vectors generated based on the feature vectors in the respective attention layers comprises:

[0011] For any self-attention layer, converting a part of the feature vectors to be learned by each head in the self-attention layer into a vector to be processed, wherein the part of the feature vector is a part of the content associated with the corresponding head in the feature vector;

[0012] For any one head, each to-be-processed vector of the remaining heads is fused with the to-be-processed vector of the head to obtain a multi-head fusion vector of the head, and a fused input vector is generated based on the multi-head fusion vector.

[0013] In one embodiment, the vector to be processed includes a query vector, a key vector and a value vector, and the step of fusing the vectors to be processed of the remaining heads with the vector to be processed of the head to obtain a multi-head fused vector of the head, and generating a fused input vector based on the multi-head fused vector includes:

[0014] Fusing the query vectors of the remaining heads with the query vector of the head to obtain a multi-head fused query vector of the head, and fusing the key vectors of the remaining heads with the key vector of the head to obtain a multi-head fused key vector of the head;

[0015] A fused input vector is generated based on the multi-head fused query vector, the multi-head fused key vector and the value vector of the head.

[0016] In one embodiment, the steps of fusing the query vectors of the remaining heads with the query vector of the head to obtain a multi-head fused query vector of the head, and fusing the key vectors of the remaining heads with the key vector of the head to obtain a multi-head fused key vector of the head include:

[0017] Performing a weighted summation on the query vectors of the remaining heads and the query vector to obtain a first summation result of the head, and performing a nonlinear transformation on the first summation result based on a preset activation function to obtain a first nonlinear representation of the head;

[0018] Performing weighted summation again on the first nonlinear representation and the query vectors of the remaining heads to obtain the multi-head fusion query vector;

[0019] Performing weighted summation on each key vector of the remaining heads and the key vector to obtain a second summation result of the head, and performing a nonlinear transformation on the second summation result based on a preset activation function to obtain a second nonlinear representation of the head;

[0020] The second nonlinear representation and the key vectors of the remaining heads are weightedly summed again to obtain the multi-head fusion key vector.

[0021] In one embodiment, the step of processing the feature representations through the optimal matching layer to obtain the matching relationship between the circuit breaker images to be matched includes:

[0022] In the optimal matching layer, a transfer matrix is ​​generated based on each feature representation, and the transfer matrix is ​​iteratively solved;

[0023] In the process of iterative solution, comparing the difference of the scaling factor between adjacent iterations with a preset threshold, wherein the scaling factor is updated in each iteration;

[0024] When the difference is smaller than the preset threshold, the current transmission matrix is ​​determined to be the optimal transmission matrix, and the matching relationship between the circuit breaker images to be matched is determined according to the optimal transmission matrix.

[0025] In one embodiment, the scaling factor includes a first scaling factor and a second scaling factor, and the step of comparing the difference between the scaling factors of adjacent iteration times with a preset threshold includes:

[0026] Compare the first scaling factor in the current iteration number with the historical first scaling factor in the previous iteration number to obtain a first difference between the first scaling factor and the historical first scaling factor;

[0027] Compare the second scaling factor in the current iteration number with the historical second scaling factor in the previous iteration number to obtain a second difference between the second scaling factor and the historical second scaling factor;

[0028] The maximum value between the first difference and the second difference is determined, and the maximum value is used as the difference of the scaling factor between the adjacent iteration times to be compared with a preset threshold.

[0029] In addition, to achieve the above purpose, the present application also proposes a circuit breaker image matching device, which is applied to a SuperGlue feature matching network, wherein the SuperGlue feature matching network includes an attention graph neural network and an optimal matching layer, wherein the attention graph neural network includes at least a plurality of alternating self-attention layers and cross-attention layers, and the circuit breaker image matching device includes:

[0030] A generating module, configured to obtain, for any one of the circuit breaker images to be matched, a feature point position and a visual descriptor of the circuit breaker image to be matched, and generate a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor;

[0031] A fusion module, used for fusing input vectors generated based on the feature vectors in respective attention layers, and transmitting the fused input vectors to the next cross attention layer, so that the attention graph neural network outputs a feature representation of the circuit breaker image to be matched;

[0032] The matching module is used to traverse each circuit breaker image to be matched to obtain each feature representation, and then process the feature representation through an optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the circuit breaker image matching method based on the improved SuperGlue as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the circuit breaker image matching method based on the improved SuperGlue as described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the circuit breaker image matching method based on the improved SuperGlue as described above.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] The present application first obtains the feature point position and visual descriptor of any one of the circuit breaker images to be matched, and generates a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor, thereby providing a reliable data basis for subsequent feature matching; in each attention layer, the input vector generated based on the feature vector is fused, and the fused input vector is transmitted to the next cross attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched, and the self-attention mechanism is used in the improved SuperGlue algorithm to perform multi-head fusion processing on the input vector, thereby enhancing the expression ability and relevance of the feature vector, and improving the accuracy and robustness of circuit breaker image feature matching in complex scenarios; after traversing each circuit breaker image to be matched to obtain each feature representation, the feature representation is processed by the optimal matching layer to obtain the matching relationship between each circuit breaker image to be matched, and the feature representation is processed by the optimal matching layer to find the accurate matching relationship from multiple circuit breaker images to be matched, thereby improving the accuracy of image matching and reducing the difficulty and error rate of manual detection.

[0038] In summary, the present application performs multi-head attention fusion processing on the input vector in the self-attention layer, so that there is information interaction between different heads, thereby avoiding the self-attention mechanism in the traditional SuperGlue algorithm from excessively focusing on its own position, resulting in a certain feature point in the data structure of the circuit breaker image easily ignoring the information of other feature points, thereby capturing the relationship between features at a finer-grained level, and achieving accurate enhancement of the expressiveness and accuracy of the feature vector even in complex scenes and changing conditions, thereby significantly improving the accuracy and robustness of matching circuit breaker images using the SuperGlue algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A flow chart of the circuit breaker image matching method embodiment 1 based on the improved SuperGlue of the present application;

[0042] Figure 2 A schematic diagram of the self-attention layer of the circuit breaker image matching method based on the improved SuperGlue provided in Example 1 of the present application;

[0043] Figure 3 A flow chart of the circuit breaker image matching method according to the second embodiment of the present application based on the improved SuperGlue;

[0044] Figure 4 A brief flowchart of the circuit breaker image matching method based on the improved SuperGlue provided in the second embodiment of the present application;

[0045] Figure 5 A schematic diagram of the first effect of the circuit breaker image matching method based on the improved SuperGlue provided in the second embodiment of the present application;

[0046] Figure 6 A schematic diagram of a second effect of the circuit breaker image matching method based on the improved SuperGlue provided in the second embodiment of the present application;

[0047] Figure 7 A third effect schematic diagram of the circuit breaker image matching method based on the improved SuperGlue provided in the second embodiment of the present application;

[0048] Figure 8 This is a schematic diagram of the module structure of a circuit breaker image matching device based on the improved SuperGlue in an embodiment of the present application;

[0049] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the circuit breaker image matching method based on the improved SuperGlue in the embodiment of the present application. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: for any one of the circuit breaker images to be matched, obtain the feature point position and visual descriptor of the circuit breaker image to be matched, and generate a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor; in each attention layer, fuse the input vector generated based on the feature vector, and transmit the fused input vector to the next cross-attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched; after traversing each circuit breaker image to be matched to obtain each feature representation, process each feature representation through the optimal matching layer to obtain the matching relationship between each circuit breaker image to be matched.

[0053] In the process of detecting circuit breaker images in the prior art, there are technical solutions that rely on image matching algorithms, such as using the open source SuperGlue algorithm to imitate the process of human eyes matching two objects through self-attention and cross-attention (when the human eye is looking for similarities between two objects, it will constantly compare the feature points between the two objects and the feature points of the objects themselves). However, in the actual use of the algorithm to train circuit breaker images and match the uploaded circuit breaker images with standard images, it was found that the algorithm has a low accuracy rate for matching complex circuit breaker images. Therefore, how to improve the accuracy of matching circuit breaker images using the SuperGlue algorithm is a problem that needs to be solved urgently.

[0054] The present application provides a solution, by performing multi-head attention fusion processing on the input vector in the self-attention layer, so that there is information interaction between different heads, thereby avoiding the self-attention mechanism in the traditional SuperGlue algorithm from excessively focusing on its own position, resulting in a certain feature point in the data structure of the circuit breaker image easily ignoring the information of other feature points, thereby capturing the relationship between features at a finer-grained level, and achieving accurate enhancement of the expression ability and accuracy of the feature vector even in complex scenes and changing conditions, thereby significantly improving the accuracy and robustness of matching circuit breaker images using the SuperGlue algorithm.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions. The following takes an electronic device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the embodiment of the present application provides a circuit breaker image matching method based on an improved SuperGlue, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the circuit breaker image matching method based on the improved SuperGlue of the present application.

[0057] In this embodiment, the circuit breaker image matching method based on the improved SuperGlue is applied to the SuperGlue feature matching network, and the SuperGlue feature matching network includes an attention graph neural network and an optimal matching layer, wherein the attention graph neural network includes at least a plurality of alternating self-attention layers and cross-attention layers, and the circuit breaker image matching method based on the improved SuperGlue includes steps S10 to S30:

[0058] Step S10, for any one of the circuit breaker images to be matched, obtaining a feature point position and a visual descriptor of the circuit breaker image to be matched, and generating a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor;

[0059] It should be noted that the feature point position refers to the coordinates of the key points in the circuit breaker image that can be stably identified and used for image matching; the visual descriptor refers to the vector that encodes the image area around the feature point, which contains the local image information of the feature point; the feature vector refers to the high-dimensional vector mapped by the attention graph neural network using the feature point encoder to multiple feature point positions and multiple visual descriptors, which is used to represent the content of the entire circuit breaker image.

[0060] It is understandable that, since stable and distinguishable information needs to be extracted from the circuit breaker image, performing step S10 can avoid the problem of inaccurate feature extraction caused by factors such as image noise and illumination changes, thereby improving the stability and reliability of feature extraction.

[0061] Exemplarily, the feature point positions and visual descriptors of each circuit breaker image to be matched are obtained. For any circuit breaker image to be matched, the feature point positions of the circuit breaker image to be matched are encoded by the feature point encoder in the attention graph neural network, and the encoded result is concatenated with the visual descriptor of the circuit breaker image to be matched to obtain a feature vector of the circuit breaker image to be matched.

[0062] Step S20, in each attention layer, the input vectors generated based on the feature vectors are fused, and the fused input vectors are transmitted to the next cross attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched;

[0063] It should be noted that the self-attention layer refers to a layer in the neural network, which can process the input feature vector accordingly so that the neural network can pay attention to the important features in the circuit breaker image; the input vector refers to the feature vector that needs to be learned by multiple heads, and is the vector representation when it is input into the self-attention layer; the cross-attention layer refers to the layer in the network that further processes the input vector, which can model the relationship between different feature vectors; the feature representation refers to the feature vector after being processed by multiple alternating self-attention layers and cross-attention layers, which can more accurately reflect the essential characteristics of the circuit breaker image.

[0064] It is understandable that since the self-attention mechanism of traditional SuperGlue will excessively focus on its own position when encoding the information of its current position, step S20 is performed. Because in SuperGlue's multi-head attention, the input feature vector will be divided into multiple heads for separate processing, so as to learn different aspects or subspaces of the feature vector through different heads, so by paying attention to the connection between different features while learning the features of each head, that is, fusing the input vector generated based on the feature vector, it is possible to introduce interaction between different heads, so that the relationship between features can be captured at a finer-grained level, thereby improving the diversity and expressiveness of the attention head, and then enabling SuperGlue to more accurately identify and align related feature points in the task of feature matching.

[0065] Exemplarily, in each self-attention layer in the attention graph neural network, a head interaction layer is added before the input of each head, wherein the head interaction layer includes convolution and activation functions to introduce interaction between different heads through the head interaction layer, that is, to fuse the input vectors.

[0066] In a feasible implementation manner, the step of fusing the input vectors generated based on the feature vectors in the respective attention layers in step S20 may include steps S21 to S22:

[0067] Step S21, for any self-attention layer, converting a part of the feature vectors to be learned by each head in the self-attention layer into a vector to be processed, wherein the part of the feature vector is a part of the content associated with the corresponding head in the feature vector;

[0068] It should be noted that a head refers to a subunit in the self-attention layer. Each head can learn different parts of the input features from different angles, thereby capturing diverse information. A partial feature vector refers to the part of the data that is segmented out from the original feature vector for processing by a specific head. A vector to be processed refers to a feature vector that has been preliminarily processed by a specific head in the self-attention layer. This vector will serve as the data basis for subsequent fusion operations.

[0069] Step S22: for any one head, fuse the to-be-processed vectors of the remaining heads with the to-be-processed vector of the head to obtain a multi-head fusion vector of the head, and generate a fused input vector based on the multi-head fusion vector.

[0070] It should be noted that the multi-head fusion vector refers to the vector obtained by merging the vectors to be processed of multiple heads. This vector integrates the feature information that needs to be learned by different heads; the fused input vector refers to the feature vector used for processing in the next layer of the network, namely the cross-attention layer, after the multi-head fusion operation. This vector contains richer information representation.

[0071] It is understandable that since it is necessary to integrate the feature information that needs to be learned by different heads in order to obtain more comprehensive and accurate image features in the learning process of any head, performing step S22 can avoid the poor generalization ability of the network caused by the failure to integrate feature information, thereby enhancing the network's adaptability and learning ability to complex images, and further improving the accuracy of image matching.

[0072] In this implementation, by adopting a multi-head self-attention mechanism and the fusion of vectors between heads, it is avoided that a single attention head excessively focuses on its own position, thereby failing to fully capture the diverse information of other feature points in the feature vector. The feature information that needs to be learned by different heads is effectively integrated, so that the neural network model can capture the relationship between features at a finer-grained level, improve the diversity and expressiveness of the attention heads, and then help the SuperGlue network model to more accurately identify and align related feature points in the task of feature matching.

[0073] In a feasible implementation manner, the vector to be processed includes a query vector, a key vector and a value vector;

[0074] It should be noted that the query vector refers to the vector used in the self-attention mechanism to extract relevant information from the input features. This vector determines which parts of the input should be focused on; the key vector refers to the vector used in the self-attention mechanism to match the query vector. This vector determines which parts of the input features will affect the extraction of the query vector; the value vector refers to the vector used for weighted summation in the self-attention mechanism. This vector contains detailed information about the input features.

[0075] The step of fusing the to-be-processed vectors of the remaining heads with the to-be-processed vector of the head in step S22 to obtain a multi-head fusion vector of the head, and generating a fused input vector based on the multi-head fusion vector may include steps S221 to S222:

[0076] Step S221, fusing the query vectors of the remaining heads with the query vector of the head to obtain a multi-head fused query vector of the head, and fusing the key vectors of the remaining heads with the key vector of the head to obtain a multi-head fused key vector of the head;

[0077] It should be noted that the multi-head fusion query vector refers to the vector obtained by fusing the query vectors of multiple attention heads, which integrates the focus of different heads on the input features; the multi-head fusion key vector refers to the vector obtained by fusing the key vectors of multiple attention heads, which integrates the correlation measurement of different heads on the input features.

[0078] It is understandable that since it is necessary to integrate the information of multiple attention heads to obtain a more comprehensive feature representation, performing step S221 can avoid the problem of feature information that may be ignored by a single head, improve the diversity of features, and enhance the model's understanding and attention to different feature parts by fusing the queries and key vectors of multiple heads.

[0079] Step S222: Generate a fused input vector based on the multi-head fused query vector, the multi-head fused key vector and the head value vector.

[0080] It can be understood that since it is necessary to generate a new feature representation based on the fused query and key vectors, as well as the original value vector, performing step S222 can avoid the problem of inaccurate feature representation due to lack of comprehensive information. By generating a fused input vector, the accuracy of the feature representation and the network model's ability to process input data are improved.

[0081] In this implementation, by performing vector fusion under the multi-head attention mechanism, the problem that a single attention head may not be able to fully capture the input feature information is avoided, and the query and key vectors of multiple heads are integrated in the multi-head attention layer to obtain richer and more comprehensive multi-head fusion query vectors and multi-head fusion key vectors, and then combined with the value vector to generate a fused input vector. This process improves the network model's comprehensive processing capabilities for input features, achieves more accurate and robust feature representation, and improves the performance and generalization capabilities of the network model in complex scenarios.

[0082] Step S30: After traversing each circuit breaker image to be matched to obtain each feature representation, the feature representation is processed by an optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched.

[0083] It should be noted that the matching relationship refers to the similarity of feature representations between different circuit breaker images, which is used to determine whether multiple circuit breaker images match.

[0084] It is understandable that, since it is necessary to determine the similarity between different circuit breaker images, step S30 is performed. By adopting the optimal matching layer to determine the matching relationship between the circuit breaker images, the inefficiency and erroneous matching of traditional matching methods in complex environments can be avoided, thereby improving the accuracy and efficiency of image matching, and further ensuring the accuracy of the inspection operation.

[0085] Exemplarily, after traversing each circuit breaker image to be matched, the optimal matching layer processes each feature representation to obtain a score matrix representing the preliminary matching relationship between each circuit breaker image to be matched, and uses the Sinkhorn algorithm (for T iterations) to find the optimal partial match based on the score matrix, thereby obtaining the matching relationship of each input circuit breaker image to be matched.

[0086] This embodiment provides a circuit breaker image matching method based on an improved SuperGlue, which performs multi-head attention fusion processing on the input vector in the self-attention layer, so that information interaction exists between different heads, thereby avoiding the self-attention mechanism in the traditional SuperGlue algorithm from excessively focusing on its own position, resulting in a certain feature point in the data structure of the circuit breaker image easily ignoring the information of other feature points, thereby capturing the relationship between features at a finer-grained level, and achieving accurate enhancement of the expression ability and accuracy of the feature vector even in complex scenes and changing conditions, thereby significantly improving the accuracy and robustness of matching circuit breaker images using the SuperGlue algorithm.

[0087] In a feasible implementation, step S221 may include steps S201 to S204:

[0088] Step S201, performing weighted summation on query vectors of other heads and the query vector to obtain a first summation result of the head, and performing a nonlinear transformation on the first summation result based on a preset activation function to obtain a first nonlinear representation of the head;

[0089] It should be noted that the preset activation function refers to a function used to introduce nonlinear factors, such as ReLU, Sigmoid or Tanh, which can help the network learn more complex features and patterns.

[0090] It can be understood that since it is necessary to make each head consider the features of other head inputs when processing features and increase the nonlinear ability of the network model when processing cross-head information, step S201 is performed. By weighted summing the query vectors of each head, it is possible to avoid not considering the features of other heads when the current head processes features. At the same time, by performing nonlinear transformation based on the activation function, it is possible to avoid the problem of poor nonlinear ability of the network model to process cross-head information, thereby improving the model's feature extraction capability for input data.

[0091] Step S202, performing weighted summation again on the first nonlinear representation and the query vectors of the remaining heads to obtain the multi-head fusion query vector;

[0092] It is understandable that since it is necessary to further integrate the query information of multiple heads to obtain a more comprehensive feature representation, performing step S202 can avoid important feature information that may be missed by a single head, and realize the fusion of feature information of multiple heads, thereby improving the comprehensiveness and accuracy of the query vector.

[0093] Step S203, performing weighted summation on each key vector of the remaining heads and the key vector to obtain a second summation result of the head, and performing a nonlinear transformation on the second summation result based on a preset activation function to obtain a second nonlinear representation of the head;

[0094] It can be understood that since it is necessary to make each head consider the features of other head inputs when processing features and increase the nonlinear ability of the network model when processing cross-head information, step S203 is performed. By weighted summing the key vectors of each head, it is possible to avoid not considering the features of other heads when the current head processes features. At the same time, by performing nonlinear transformation based on the activation function, it is possible to avoid the problem of poor nonlinear ability of the network model to process cross-head information, thereby improving the model's feature extraction capability for input data.

[0095] Step S204: performing weighted summation on the second nonlinear representation and the key vectors of the remaining heads again to obtain the multi-head fusion key vector.

[0096] It is understandable that since it is necessary to further integrate the query information of multiple heads to obtain a more comprehensive feature representation, performing step S204 can avoid important feature information that may be missed by a single head, and realize the fusion of feature information of multiple heads, thereby improving the comprehensiveness and accuracy of the key vector.

[0097] For example, to help understand the implementation process of this embodiment, please refer to Figure 2, by introducing a head interaction layer into the traditional self-attention layer (including linear transformation, scaled dot product attention, function, linear transformation, etc. in sequence), weighted summation is performed on each query vector (Q) and each key vector (K) through the 1x1 convolution in the head interaction layer, and the corresponding summation result is nonlinearly transformed through a preset activation function (ReLU) to obtain the corresponding nonlinear representation. The obtained nonlinear representation is weighted summed again based on the 1x1 convolution. After that, based on the multi-head fused query vector and multi-head fused key vector obtained by the weighted summation again, the fused input vector is generated based on the head value vector (V) through scaled dot product attention, function, linear transformation, etc.

[0098] In this implementation, by performing weighted summation and nonlinear transformation, the problem that linear processing cannot capture complex relationships between features is avoided, and in the multi-head attention mechanism, the query vector and the key vector are first weighted summed and nonlinearly transformed to obtain the first and second nonlinear representations, and then weighted summed with the vectors of other heads to obtain the multi-head fusion query vector and the multi-head fusion key vector. This process not only enhances the network model's ability to express complex relationships between features, but also improves the generalization ability and accuracy of the network model when processing multi-dimensional features, thereby achieving a more refined and comprehensive feature representation, and improving the performance and robustness of the network model in complex scenarios.

[0099] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 The step of processing the feature representations through the optimal matching layer in step S30 to obtain the matching relationship between the circuit breaker images to be matched may also include steps S31 to S33:

[0100] Step S31, generating a transfer matrix based on each feature representation in the optimal matching layer, and iteratively solving the transfer matrix;

[0101] It should be noted that the transfer matrix refers to a matrix used to represent the feature representation matching relationship between the circuit breaker images to be matched, and the matrix can determine the best match between each pair of image features through an optimization algorithm.

[0102] It is understandable that, since it is necessary to find the best feature correspondence between the circuit breaker images to be matched, step S31 is performed to avoid erroneous matching or inefficient matching that may be caused by direct matching, and to find the best match between feature representations through iterative optimization, thereby improving the accuracy and efficiency of matching.

[0103] Exemplarily, in the optimal matching layer, a score matrix is ​​constructed based on each feature representation, and the last column / row of the score matrix is ​​set to dustbins to obtain a cost matrix. Then, the obtained cost matrix is ​​multiplied by the element of the negative regularization parameter λ, and an element-by-element exponential function is applied to obtain a transfer matrix, that is, the transfer matrix K is calculated by the following formula:

[0104] K=e -λM

[0105] Where M is the cost matrix. Then K is solved iteratively.

[0106] Step S32, during the iterative solution process, comparing the difference between the scaling factors of adjacent iterations with a preset threshold, wherein the scaling factor is updated during each iteration;

[0107] It should be noted that the scaling factor refers to a value used to adjust the transfer matrix during the iterative solution process. This value is updated with each iteration and can be used to optimize the matching quality between feature representations.

[0108] It is understandable that since redundant calculations often exist in existing iterative solution algorithms, step S32 is performed. By comparing the difference in scaling factors between adjacent iterations in the iterative solution process and a preset threshold, the convergence degree of the Sinkhorn algorithm can be determined, and unnecessary calculation problems can be avoided, thereby significantly reducing the calculation time (especially when the algorithm converges quickly).

[0109] For example, in the iterative solution process, there is a scaling factor u∈R d1 ,v∈R d2 , d1 represents the dimension of r distribution, d2 represents the dimension of c distribution, where r distribution and c distribution define the rows and columns of the transmission matrix K, as well as the constraints. And u and v are initialized as:

[0110]

[0111] Then use the Sinkhorn algorithm to iteratively solve u and v:

[0112]

[0113] During the iterative solution process, the difference between the scaling factors u and v between adjacent iterations is compared with a preset threshold.

[0114] In a feasible implementation manner, the scaling factor includes a first scaling factor and a second scaling factor, and the step of comparing the difference in scaling factors between adjacent iteration times with a preset threshold in step S32 may include steps S321 to S323:

[0115] Step S321, compare the first scaling factor in the current iteration number with the historical first scaling factor in the previous iteration number to obtain a first difference between the first scaling factor and the historical first scaling factor; the historical first scaling factor refers to the scaling factor with the same function used in the previous iteration number, which is the result of the previous iteration.

[0116] It should be noted that the first scaling factor refers to the scaling factor used to adjust the transfer matrix in the current iteration, which reflects the matching degree between feature representations in the current iteration;

[0117] Step S322, comparing the second scaling factor in the current iteration number with the historical second scaling factor in the previous iteration number to obtain a second difference between the second scaling factor and the historical second scaling factor;

[0118] It should be noted that the second scaling factor refers to a scaling factor corresponding to the first scaling factor, but in a different matching relationship or a different feature dimension; the historical second scaling factor also refers to the corresponding scaling factor in the previous iteration number.

[0119] Step S323: determine the maximum value between the first difference and the second difference, and use the maximum value as the difference of the scaling factor between the adjacent iteration times to compare with a preset threshold.

[0120] It is understandable that in order to ensure that the change of the scaling factor is minimal during the entire iteration process, step S323 is performed to avoid premature cessation of iteration due to small differences in local scaling factors and large differences in overall scaling factors, thereby achieving the effect of global matching optimization.

[0121] In this implementation, by comparing the differences between different scaling factors in the current iteration and the previous iteration, and determining whether the algorithm has converged based on the maximum value of each difference, the problem of premature convergence of the algorithm due to the change of a single scaling factor being insufficient to reflect the overall iterative convergence is avoided, and the effect of judging the iterative convergence is achieved by monitoring the changes in multiple key scaling factors. This ensures that during the iteration process, even in the case of multi-dimensional feature matching, it is possible to accurately judge whether the iteration is close to the optimal solution, thereby avoiding premature stopping or excessive iteration that may occur in the iterative process, and improving the stability of the algorithm and the accuracy of matching.

[0122] Step S33, when the difference is less than the preset threshold, determining the current transmission matrix as the optimal transmission matrix, and determining the matching relationship between each to-be-matched circuit breaker image according to the optimal transmission matrix.

[0123] It should be noted that the optimal transmission matrix refers to a transmission matrix that can optimally represent the matching relationship between the circuit breaker images to be matched after iterative solution. This matrix reflects the best matching scheme between the matching circuit breaker images.

[0124] It can be understood that since it is necessary to determine when to stop iteration and use the current transmission matrix as the final matching result, step S33 is performed to avoid redundant calculation problems caused by excessive iterations and suboptimal matching results caused by insufficient iterations, and to achieve the determination of the optimal transmission matrix when the iterations are sufficiently converged, thereby ensuring that the final matching relationship meets the performance requirements while improving the efficiency of determining the matching results.

[0125] Exemplarily, the early stopping Sinkhorn algorithm proposed in this embodiment is implemented by the following pseudo code:

[0126]

[0127]

[0128] Among them, IMPROVEDSINKHORN is the name of a function implemented by an improved Sinkhorn algorithm, which receives parameters u, v (i.e., two scaling factors), K (transmission matrix), kmax (maximum number of iterations), and ε (preset threshold). r is a vector of source distribution, representing the data or resource transmission from here, which can be a vector of all 1s, representing the total weight of the source point; c is a vector of target distribution, representing the location where the data or resource needs to be transmitted, which can also be a vector of all 1s, representing the total weight of the target point. preu u and prev v Initialized to None, they are used to store the u and v values ​​of the previous iteration in order to check convergence. The iterative process of this pseudocode is: Use a for loop to iterate, with a maximum of k iterations max times, and in each iteration, new u and v values ​​are calculated; u new It is calculated by dividing r by the product of K times v, where the division is performed element-wise; v new By dividing c by the transpose of K and multiplying it by u new The result is calculated. If preu u and prev v If it is not None (i.e. not the first iteration), check u new and v newIs the difference between their previous values ​​less than the threshold ε. The difference here is measured by the infinity norm (maximum absolute difference). If the difference is less than ε, the algorithm is considered to have converged and exit the loop. After the iteration is completed, the final optimal transmission matrix P* is calculated, which is obtained by multiplying the diagonal matrices diag(u) and diag(v) with the transmission matrix K. C* is the optimal transmission cost, which is obtained by calculating the Frobenius inner product of the final transmission matrix P* and the original cost matrix M.

[0129] In this embodiment, by monitoring the change in the scaling factor during the iteration process, that is, the difference in the scaling factor between adjacent iterations, the redundant calculation problem existing in the traditional Sinkhorn algorithm is avoided, that is, the algorithm often has converged but still continues to perform iterative calculations, and the algorithm iteration is stopped in advance according to the degree of convergence, thereby speeding up the solution speed of the Sinkhorn algorithm.

[0130] For example, in order to help understand the implementation process of the circuit breaker image matching method based on the improved SuperGlue obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 4 , Figure 4 A brief flowchart of a circuit breaker image matching method based on an improved SuperGlue is provided. Specifically:

[0131] In the improved SuperGlue feature matching network of the present application, the feature vectors of each circuit breaker image to be matched are first generated, and the input vectors are fused in the self-attention layer to obtain the feature representations of the output of the attention graph neural network. A transmission matrix is ​​generated based on the feature representations, and the early stopping Sinkhorn algorithm of the present application is used to iteratively solve the transmission matrix, that is, the difference in scaling factors between adjacent iterations is monitored during the iterative solution process, so as to stop the iteration when the difference is less than a preset threshold, obtain the optimal transmission matrix, and determine the matching relationship of each circuit breaker image to be matched according to the optimal transmission matrix.

[0132] Regarding the head cross layer (HeadCrossLayer) introduced in the traditional self-attention in this application, in the ablation experiment of the head cross layer, whether the homography image corresponding to each test set image is generated by the DLT (Direct Linear Transform) method or the RandomHomography method, the head cross layer can bring good performance improvement, as shown in Table 1 below:

[0133]

[0134] Table 1

[0135] As can be seen from Table 1, compared with the SuperGlue+DLT method, the AUC (Area Under Curve) of the SuperGlue+HeadCrossLayer+DLT method with thresholds of 10 and 25 has increased by 1.58 and 8.48 percentage points respectively. The recall rate has been significantly improved by 38.25 percentage points. Compared with SuperGlue+Homography, adding a head interaction layer to SuperGlue+HeadCrossLayer+Homography also brings significant results, with the AUC with a threshold of 5 increasing by 10.57 percentage points and the AUC with a threshold of 10 increasing by 7.72 percentage points. In particular, the recall rate has been significantly improved by 38.25 percentage points.

[0136] For the early stop mechanism (Early Stop Sinkhorn) introduced in the traditional Sinkhorn algorithm in this application, the indoor dataset and COCO dataset are used to conduct early stop Sinkhorn ablation experiments, as shown in Table 2 below:

[0137]

[0138] Table 2

[0139] In Table 2, the precision, recall, and AUC of the three thresholds of Superglue+Sinkhorn and Superglue+EarlyStopSinkhorn are the same. Therefore, the early stop Sinkhorn algorithm does not affect the performance of the model in terms of accuracy. At the same time, in the indoor dataset and COCO dataset, the early stop Sinkhorn algorithm is verified to improve the speed of the Sinkhorn algorithm. Please refer to Figure 5 and Figure 6 , the horizontal axis P is the number of image pairs inferred by SuperGlue, and the vertical axis S is the inference time for each pair of images. From the two figures, we can see that the early stopping Sinkhorn algorithm (superglue_sinkhorn) can improve the inference speed compared with the classic Sinkhorn algorithm (superglue). In particular, the early stopping Sinkhorn algorithm is about 12.5 percentage points higher than the classic Sinkhorn algorithm at the beginning. As the number of image pairs processed by the model increases, the difference between the two algorithms decreases, and finally converges to around 0.30s. It can be concluded that the improved Sinkhorn algorithm can speed up convergence.

[0140] After using the improved SuperGlue feature matching network, it is used in the matching task of two different circuit breaker images to be matched, and the obtained Figure 7The green line indicates a feature point match that is very easy to judge, the blue line indicates a feature point match that is moderately judged, and the red line indicates a feature point match that is difficult to judge. Figure 7 The left image in (a) is an image of the entire circuit breaker panel taken under dark conditions, and the right image is an image of part of the circuit breaker taken under normal lighting conditions during the day. The improved network not only successfully matches the feature points of the pressure plate in the left and right images, but also matches the indicator light, the corresponding nameplate of the pressure plate, etc. It can be seen that the matching effect is good and can complete the matching task well; Figure 7 (b) and (d) show that if the circuit breaker image is complete, the lighting conditions are good, and the angle change is small, the improved network can bring more accurate matching results and cover most of the details in the image. Figure 7 The indicator lights, displays, display buttons, pressure plates, pressure plate corresponding nameplates, and knobs of (b) and (d) can achieve an excellent matching effect; Figure 7 (c) shows another extreme match. The right image in (c) accounts for less than one-fifth of the circuit breaker image, with only some pressure plates, indicator lights, knobs, etc., and there are black areas in the image as interference. The left image is an image of the circuit breaker taken under normal operating conditions and lighting conditions. The improved network can still determine the matching relationship between the left and right images. From the above matching result images, it can be seen that the improved SuperGlue has the ability to overcome problems such as lighting conditions, angle changes, and partial features, and can better complete the circuit breaker image matching task.

[0141] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the circuit breaker image matching method based on the improved SuperGlue of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0142] This application also provides a circuit breaker image matching device, please refer to Figure 8 The circuit breaker image matching device is applied to a SuperGlue feature matching network, wherein the SuperGlue feature matching network includes an attention graph neural network and an optimal matching layer, wherein the attention graph neural network includes at least a plurality of alternating self-attention layers and cross-attention layers, and the circuit breaker image matching device includes:

[0143] A generating module 10 is used for obtaining, for any one of the circuit breaker images to be matched, a feature point position and a visual descriptor of the circuit breaker image to be matched, and generating a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor;

[0144] A fusion module 20 is used to fuse the input vectors generated based on the feature vectors in the respective attention layers, and transmit the fused input vectors to the next cross attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched;

[0145] The matching module 30 is used to obtain each feature representation by traversing each circuit breaker image to be matched, and then process the feature representation through an optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched.

[0146] Optionally, the fusion module 20 is further used for:

[0147] For any self-attention layer, converting a part of the feature vectors to be learned by each head in the self-attention layer into a vector to be processed, wherein the part of the feature vector is a part of the content associated with the corresponding head in the feature vector;

[0148] For any one head, each to-be-processed vector of the remaining heads is fused with the to-be-processed vector of the head to obtain a multi-head fusion vector of the head, and a fused input vector is generated based on the multi-head fusion vector.

[0149] Optionally, the vector to be processed includes a query vector, a key vector and a value vector, and the fusion module 20 is further used for:

[0150] Fusing the query vectors of the remaining heads with the query vector of the head to obtain a multi-head fused query vector of the head, and fusing the key vectors of the remaining heads with the key vector of the head to obtain a multi-head fused key vector of the head;

[0151] A fused input vector is generated based on the multi-head fused query vector, the multi-head fused key vector and the value vector of the head.

[0152] Optionally, the fusion module 20 is further used for:

[0153] Performing a weighted summation on the query vectors of the remaining heads and the query vector to obtain a first summation result of the head, and performing a nonlinear transformation on the first summation result based on a preset activation function to obtain a first nonlinear representation of the head;

[0154] Performing weighted summation again on the first nonlinear representation and the query vectors of the remaining heads to obtain the multi-head fusion query vector;

[0155] Performing weighted summation on each key vector of the remaining heads and the key vector to obtain a second summation result of the head, and performing a nonlinear transformation on the second summation result based on a preset activation function to obtain a second nonlinear representation of the head;

[0156] The second nonlinear representation and the key vectors of the remaining heads are weightedly summed again to obtain the multi-head fusion key vector.

[0157] Optionally, the matching module 30 is further used for:

[0158] In the optimal matching layer, a transfer matrix is ​​generated based on each feature representation, and the transfer matrix is ​​iteratively solved;

[0159] In the process of iterative solution, comparing the difference of the scaling factor between adjacent iterations with a preset threshold, wherein the scaling factor is updated in each iteration;

[0160] When the difference is smaller than the preset threshold, the current transmission matrix is ​​determined to be the optimal transmission matrix, and the matching relationship between the circuit breaker images to be matched is determined according to the optimal transmission matrix.

[0161] Optionally, the scaling factor includes a first scaling factor and a second scaling factor, and the matching module 30 is further configured to:

[0162] Compare the first scaling factor in the current iteration number with the historical first scaling factor in the previous iteration number to obtain a first difference between the first scaling factor and the historical first scaling factor;

[0163] Compare the second scaling factor in the current iteration number with the historical second scaling factor in the previous iteration number to obtain a second difference between the second scaling factor and the historical second scaling factor;

[0164] The maximum value between the first difference and the second difference is determined, and the maximum value is used as the difference of the scaling factor between the adjacent iteration times to be compared with a preset threshold.

[0165] The circuit breaker image matching device provided by the present application adopts the circuit breaker image matching method based on the improved SuperGlue in the above embodiment, which can solve the technical problems of electronics. Compared with the prior art, the beneficial effects of the circuit breaker image matching device provided by the present application are the same as the beneficial effects of the circuit breaker image matching method based on the improved SuperGlue provided in the above embodiment, and the other technical features in the circuit breaker image matching device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0166] The present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the circuit breaker image matching method based on the improved SuperGlue in the above-mentioned embodiment 1.

[0167] Reference below Fig. 9 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0168] like Fig. 9 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0169] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0170] The electronic device provided by the present application adopts the circuit breaker image matching method based on the improved SuperGlue in the above embodiment, which can solve the technical problem of how to improve the accuracy of matching circuit breaker images using the SuperGlue algorithm. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the circuit breaker image matching method based on the improved SuperGlue provided in the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0171] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the circuit breaker image matching method based on the improved SuperGlue in the above-mentioned embodiment.

[0172] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system or device.

[0173] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: for any one of the circuit breaker images to be matched, obtain the feature point position and visual descriptor of the circuit breaker image to be matched, and generate a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor; in each attention layer, fuse the input vector generated based on the feature vector, and transmit the fused input vector to the next cross-attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched; after traversing each circuit breaker image to be matched to obtain each feature representation, process each feature representation through the optimal matching layer to obtain the matching relationship between each circuit breaker image to be matched.

[0174] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned circuit breaker image matching method based on the improved SuperGlue, and can solve the technical problem of how to improve the accuracy of matching circuit breaker images using the SuperGlue algorithm. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the circuit breaker image matching method based on the improved SuperGlue provided in the above-mentioned embodiment, and will not be repeated here.

[0175] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the circuit breaker image matching method based on the improved SuperGlue as described above.

[0176] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy of matching circuit breaker images using the SuperGlue algorithm. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the circuit breaker image matching method based on the improved SuperGlue provided in the above embodiment, which will not be elaborated here.

Claims

1. A circuit breaker image matching method based on improved SuperGlue, characterized in that: Applied to a SuperGlue feature matching network, the SuperGlue feature matching network includes an attention graph neural network and an optimal matching layer, wherein the attention graph neural network includes at least a plurality of alternating self-attention layers and cross-attention layers, and the circuit breaker image matching method based on the improved SuperGlue includes: For any one of the to-be-matched circuit breaker images, obtain a feature point position and a visual descriptor of the to-be-matched circuit breaker image, and generate a feature vector of the to-be-matched circuit breaker image according to the feature point position and the visual descriptor; In each attention layer, the input vectors generated based on the feature vectors are fused, and the fused input vectors are transmitted to the next cross attention layer, so that the attention graph neural network outputs the feature representation of the circuit breaker image to be matched; After traversing each circuit breaker image to be matched to obtain each feature representation, the feature representation is processed by the optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched.

2. The circuit breaker image matching method based on improved SuperGlue according to claim 1, characterized in that: The step of fusing the input vectors generated based on the feature vectors in the respective attention layers comprises: For any self-attention layer, converting a part of the feature vectors to be learned by each head in the self-attention layer into a vector to be processed, wherein the part of the feature vector is a part of the content associated with the corresponding head in the feature vector; For any one head, each to-be-processed vector of the remaining heads is fused with the to-be-processed vector of the head to obtain a multi-head fusion vector of the head, and a fused input vector is generated based on the multi-head fusion vector.

3. The circuit breaker image matching method based on improved SuperGlue according to claim 2, characterized in that: The vector to be processed includes a query vector, a key vector and a value vector. The step of fusing the vectors to be processed of the remaining heads with the vector to be processed of the head to obtain a multi-head fused vector of the head, and generating a fused input vector based on the multi-head fused vector includes: Fusing the query vectors of the remaining heads with the query vector of the head to obtain a multi-head fused query vector of the head, and fusing the key vectors of the remaining heads with the key vector of the head to obtain a multi-head fused key vector of the head; A fused input vector is generated based on the multi-head fused query vector, the multi-head fused key vector and the value vector of the head.

4. The circuit breaker image matching method based on improved SuperGlue according to claim 3, characterized in that: The steps of fusing the query vectors of the remaining heads with the query vector of the head to obtain a multi-head fused query vector of the head, and fusing the key vectors of the remaining heads with the key vector of the head to obtain a multi-head fused key vector of the head include: Performing a weighted summation on the query vectors of the remaining heads and the query vector to obtain a first summation result of the head, and performing a nonlinear transformation on the first summation result based on a preset activation function to obtain a first nonlinear representation of the head; Performing weighted summation again on the first nonlinear representation and the query vectors of the remaining heads to obtain the multi-head fusion query vector; Performing weighted summation on each key vector of the remaining heads and the key vector to obtain a second summation result of the head, and performing a nonlinear transformation on the second summation result based on a preset activation function to obtain a second nonlinear representation of the head; The second nonlinear representation and the key vectors of the remaining heads are weightedly summed again to obtain the multi-head fusion key vector.

5. The circuit breaker image matching method based on improved SuperGlue according to claim 1, characterized in that: After traversing each circuit breaker image to be matched to obtain each feature representation, the step of processing each feature representation through the optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched includes: In the optimal matching layer, a transfer matrix is ​​generated based on each feature representation, and the transfer matrix is ​​iteratively solved; In the process of iterative solution, comparing the difference of the scaling factor between adjacent iterations with a preset threshold, wherein the scaling factor is updated in each iteration; When the difference is smaller than the preset threshold, the current transmission matrix is ​​determined to be the optimal transmission matrix, and the matching relationship between the circuit breaker images to be matched is determined according to the optimal transmission matrix.

6. The circuit breaker image matching method based on improved SuperGlue according to claim 5, characterized in that: The scaling factor includes a first scaling factor and a second scaling factor, and the step of comparing the difference between the scaling factors of adjacent iteration times with a preset threshold includes: Compare the first scaling factor in the current iteration number with the historical first scaling factor in the previous iteration number to obtain a first difference between the first scaling factor and the historical first scaling factor; Compare the second scaling factor in the current iteration number with the historical second scaling factor in the previous iteration number to obtain a second difference between the second scaling factor and the historical second scaling factor; The maximum value between the first difference and the second difference is determined, and the maximum value is used as the difference of the scaling factor between the adjacent iteration times to be compared with a preset threshold.

7. A circuit breaker image matching device, characterized in that: The circuit breaker image matching device is applied to a SuperGlue feature matching network, wherein the SuperGlue feature matching network includes an attention graph neural network and an optimal matching layer, wherein the attention graph neural network includes at least a plurality of alternating self-attention layers and cross-attention layers, and the circuit breaker image matching device includes: A generating module, configured to obtain, for any one of the circuit breaker images to be matched, a feature point position and a visual descriptor of the circuit breaker image to be matched, and generate a feature vector of the circuit breaker image to be matched according to the feature point position and the visual descriptor; A fusion module, used for fusing input vectors generated based on the feature vectors in respective attention layers, and transmitting the fused input vectors to the next cross attention layer, so that the attention graph neural network outputs a feature representation of the circuit breaker image to be matched; The matching module is used to traverse each circuit breaker image to be matched to obtain each feature representation, and then process the feature representation through an optimal matching layer to obtain a matching relationship between each circuit breaker image to be matched.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the circuit breaker image matching method based on the improved SuperGlue as claimed in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the circuit breaker image matching method based on the improved SuperGlue are implemented as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the circuit breaker image matching method based on the improved SuperGlue are implemented as claimed in any one of claims 1 to 6.

Images