Image matching method, positioning method, device, equipment and storage medium

Abstract

Embodiments of the present application provide image matching methods, positioning methods, devices, equipment and storage media, and relate to the technical field of artificial intelligence. The image matching method extracts features of each query image in a query image sequence to obtain an image feature sequence, obtains a residual fusion feature sequence according to the image feature sequence, calculates the similarity between the residual fusion feature sequence and at least one matching feature sequence to obtain a candidate image sequence, then calculates a similarity matrix according to the image feature sequence and the candidate image sequence, and selects a matching image sequence according to the similarity matrix to obtain a target image. The residual feature processing module extracts residual features from the image feature sequence, and fuses the residual features to obtain a residual fusion feature sequence. In combination with prior knowledge of the image, the residual features are used to replace the image features, so as to eliminate the influence of appearance changes on the image matching similarity and improve the search accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to image matching methods, localization methods, devices, equipment, and storage media. Background Technology

[0002] Currently, image matching is being used more and more widely, enabling applications such as target recognition, result search, and location information acquisition. For example, intelligent robots need to rely on image matching to complete tasks in production operations, construction operations, or hazardous environments.

[0003] In related technologies, image matching often involves using deep learning models to directly extract features from image sequences and match them with features from pre-stored images in a database. However, in time-series scenarios, the appearance of the obtained images changes over time. Therefore, as the number of image matching attempts increases, the error in the matching results also increases, reducing the accuracy of matching the current image in the database. For example, intelligent robots use visual position recognition devices to maintain a map information database, matching the robot's current environment image with images in the database for localization. After a long period of operation, changes in appearance, such as seasons, lighting, and weather, can significantly affect the image content, causing a large difference between the currently acquired image information and the image information stored in the database. This leads to drift in the robot's localization information on the navigation map, resulting in significant errors in robot localization. Summary of the Invention

[0004] The main objective of this application is to provide an image matching method, positioning method, apparatus, device, and storage medium that can improve the accuracy of image matching.

[0005] To achieve the above objectives, a first aspect of this application proposes an image matching method, comprising:

[0006] Obtain a query image sequence, the query image sequence including at least one query image based on a time series;

[0007] Each query image is input into the image feature extraction module for feature extraction to obtain an image feature sequence;

[0008] The image feature sequence is input into the residual feature processing module for residual feature extraction to obtain the residual fusion feature sequence;

[0009] The similarity between the residual fused feature sequence and at least one matching feature sequence is calculated to obtain the similarity value between the fused feature sequence and each of the matching feature sequences.

[0010] Multiple matching feature sequences are selected based on the similarity values ​​to obtain candidate image sequences;

[0011] Calculate a similarity matrix based on the image feature sequence and the candidate image sequence;

[0012] The target image is obtained by selecting the matching image sequence corresponding to the matching feature sequence based on the similarity matrix.

[0013] In some embodiments, the image feature extraction module includes at least one convolutional unit, and each convolutional unit includes at least one convolutional layer, an activation layer, and a pooling layer connected in sequence.

[0014] The step of inputting each query image into the image feature extraction module for feature extraction to obtain an image feature sequence includes: sequentially inputting the query images into the convolution unit for convolution, activation and pooling operations to obtain an image feature sequence; the image feature sequence includes at least one image feature, and each query image corresponds to one image feature.

[0015] In some embodiments, the residual feature processing module includes: a residual feature extraction module and a residual feature fusion module; the step of inputting the image feature sequence into the residual feature processing module for residual feature extraction to obtain a residual fused feature sequence includes:

[0016] The image feature sequence is input into the residual feature extraction module for residual feature extraction to obtain the residual feature sequence;

[0017] The residual feature sequence is input into the residual feature fusion module for residual feature fusion to obtain the residual fused feature sequence.

[0018] In some embodiments, the step of inputting the image feature sequence into the residual feature extraction module for residual feature extraction to obtain a residual feature sequence includes:

[0019] Obtain the first sliding window feature sequence of the image feature sequence within the first preset time window;

[0020] Obtain the second sliding window feature sequence of the image feature sequence within the second preset time window;

[0021] The first residual feature is calculated based on the first sliding window feature sequence;

[0022] The second residual feature is calculated based on the second sliding window feature sequence;

[0023] The residual feature sequence is calculated based on the first residual feature and the second residual feature.

[0024] In some embodiments, the step of inputting the residual feature sequence into the residual feature fusion module for residual feature fusion to obtain a residual fused feature sequence includes:

[0025] The first residual feature is input into the residual feature fusion module to obtain the first fused output feature;

[0026] The first fused output feature and the second residual feature are input into the residual feature fusion module to obtain the residual fused feature sequence.

[0027] In some embodiments, before inputting the image feature sequence into the residual feature processing module for residual feature extraction to obtain the residual fused feature sequence, the method further includes: training the residual feature processing module, the training process including:

[0028] Obtain a training sample set, which includes: training image samples and fusion labels, wherein the fusion labels include a first label and a second label;

[0029] The training image samples are sequentially input into the residual feature extraction module and the residual feature fusion module to obtain the residual fusion prediction sequence;

[0030] Calculate the first error value between the residual fusion prediction sequence and the first label;

[0031] Calculate the second error value between the residual fusion prediction sequence and the second label;

[0032] The loss value is calculated based on the first error value and the second error value;

[0033] The model parameters of the residual feature processing module are adjusted based on the loss value.

[0034] In some embodiments, the candidate image sequence includes at least one matching feature sequence, which is obtained by inputting the matching image into the image feature extraction module for feature extraction;

[0035] The step of calculating the similarity matrix based on the image feature sequence and the candidate image sequence includes:

[0036] Calculate the matching similarity matrix between the image feature sequence and each of the matching feature sequences;

[0037] The similarity matrix is ​​obtained by combining the matching similarity matrix.

[0038] In some embodiments, calculating the matching similarity matrix between the image feature sequence and each of the matching feature sequences includes:

[0039] Each feature value in the image feature sequence is matched one by one with each feature value in the matching feature sequence to obtain the row vector of the matching similarity matrix;

[0040] The matching similarity matrix is ​​constructed based on the row vectors.

[0041] In some embodiments, the step of selecting a matching image sequence corresponding to the matching feature sequence based on the similarity matrix to obtain the target image includes:

[0042] Calculate the mean of the values ​​on the diagonal of each of the matching similarity matrices;

[0043] Select the matching feature sequence corresponding to the matching similarity matrix with the largest mean;

[0044] Obtain the matching image sequence corresponding to the matching feature sequence as the target image sequence;

[0045] The target image is obtained from the target image sequence.

[0046] To achieve the above objectives, a second aspect of this application provides a positioning method, comprising:

[0047] Obtain multiple environmental images as a query image sequence;

[0048] Perform the image matching method as described in any of the first aspects to obtain the target image based on the query image sequence;

[0049] The target object is located based on the target image and the preset image-position mapping relationship.

[0050] To achieve the above objectives, a third aspect of this application provides an image matching apparatus, comprising:

[0051] An acquisition unit is used to acquire a query image sequence, the query image sequence including at least one query image based on a time series.

[0052] An image feature extraction unit is used to input each query image into the image feature extraction module for feature extraction to obtain an image feature sequence;

[0053] The residual feature extraction unit is used to input the image feature sequence into the residual feature processing module for residual feature extraction to obtain a residual fusion feature sequence;

[0054] The first similarity calculation unit is used to calculate the similarity between the residual fused feature sequence and at least one matching feature sequence to obtain the similarity value between the fused feature sequence and each of the matching feature sequences.

[0055] A candidate image sequence selection unit is used to select multiple matching feature sequences based on the similarity value to obtain a candidate image sequence;

[0056] The second similarity calculation unit is used to calculate a similarity matrix based on the image feature sequence and the candidate image sequence;

[0057] The target image sequence selection unit is used to select the matching image sequence corresponding to the matching feature sequence according to the similarity matrix to obtain the target image.

[0058] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first or second aspect.

[0059] To achieve the above objectives, a fifth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first or second aspect.

[0060] The image matching method, localization method, apparatus, device, and storage medium proposed in this application extract features from each query image in the acquired query image sequence to obtain an image feature sequence. Then, the image feature sequence is input into a residual feature processing module for residual feature extraction, and the residual features are fused to obtain a residual fused feature sequence. The residual fused feature sequence is then compared with at least one matching feature sequence to obtain a similarity value between the fused feature sequence and each matching feature sequence. Multiple matching feature sequences are then selected based on the similarity values ​​to obtain a candidate image sequence. Next, a similarity matrix is ​​calculated based on the image feature sequence and the candidate image sequences. Finally, the matching image sequence corresponding to the matching feature sequence is selected based on the similarity matrix to obtain the target image. This application embodiment extracts residual features from the image feature sequence using a residual feature processing module and fuses the residual features to obtain a residual fused feature sequence. By combining prior knowledge of the image and using residual features to replace image features, the influence of appearance changes on image matching similarity is eliminated, thereby improving search accuracy. Attached Figure Description

[0061] Figure 1 This is a flowchart of the image matching method provided in an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the image feature extraction module of the image matching method provided in the embodiment of the present invention.

[0063] Figure 3 This is a schematic diagram of the residual feature processing module of the image matching method provided in this embodiment of the invention.

[0064] Figure 4 yes Figure 1The flowchart of step S130.

[0065] Figure 5 yes Figure 4 The flowchart of step S131.

[0066] Figure 6 This is a schematic diagram of the residual feature fusion module of the image matching method provided in the embodiment of the present invention.

[0067] Figure 7 yes Figure 4 The flowchart for step S132 in the process.

[0068] Figure 8 This is a flowchart of an image matching method provided in another embodiment of the present invention.

[0069] Figure 9 This is a flowchart of the pre-trained residual feature processing module of the image matching method provided in another embodiment of the present invention.

[0070] Figure 10 yes Figure 1 The flowchart for step S160 in the process.

[0071] Figure 11 This is a schematic diagram of the matching similarity matrix of an image matching method provided in another embodiment of the present invention.

[0072] Figure 12 yes Figure 1 The flowchart of step S170 in the process.

[0073] Figure 13 This is a flowchart of an image matching method provided in another embodiment of the present invention.

[0074] Figure 14 This is a schematic diagram of a scenario for a positioning method provided in an embodiment of the present invention.

[0075] Figure 15 This is a structural block diagram of an image matching device provided in another embodiment of the present invention.

[0076] Figure 16 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0078] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0080] First, let's clarify some of the terms used in this invention:

[0081] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0082] Deep learning involves learning the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. The ultimate goal is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous technologies. Deep learning has also yielded significant advancements in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thought, solving many complex pattern recognition problems and significantly advancing artificial intelligence technologies.

[0083] Feature extraction involves reducing the dimensionality of the original input data or recombining the original features for later use. Simply put, it has two functions: reducing data dimensionality and organizing existing data features.

[0084] Currently, image matching is being used more and more widely, enabling applications such as target recognition, result search, and location information acquisition. For example, intelligent robots are becoming increasingly common, assisting people in related tasks, such as production operations, construction work, or operations in hazardous environments. Completing these tasks requires intelligent robots to have precise positioning capabilities.

[0085] However, in related technologies, image matching often involves using deep learning models to directly extract features from image sequences and match them with features pre-stored in a database. But in time-series scenarios, the appearance of the obtained images changes over time. Therefore, as the number of image matching attempts increases, the error in the matching result also increases, reducing the accuracy of matching the current image in the database. For example, intelligent robots use visual position recognition devices to maintain a map information database. This database binds and stores geographic information and image sequences from the map. The visual position recognition device extracts features with scene location information from the images and matches the robot's current environment image with images in the database for localization. After a long period of operation, changes in appearance, such as seasons, lighting, and weather, can significantly affect the image content, causing a large difference between the currently acquired image information and the image information stored in the database. This causes the robot's localization information to drift in the navigation map, leading to significant errors in robot localization.

[0086] Based on this, embodiments of the present invention provide an image matching method, a localization method, an apparatus, a device, and a storage medium. The residual feature processing module extracts residual features from the image feature sequence and fuses the residual features to obtain a residual fused feature sequence. Combined with prior knowledge of the image, the residual features are used to replace the image features in order to eliminate the influence of appearance changes on image matching similarity and improve search accuracy.

[0087] The present invention provides an image matching method, a positioning method, an apparatus, a device, and a storage medium, which are specifically described through the following embodiments. First, the image matching method in the embodiments of the present invention is described.

[0088] This invention can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0089] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0090] The image matching method provided in this invention relates to the field of artificial intelligence technology, and particularly to the field of data mining technology. The image matching method provided in this invention can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as an image matching client; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The image matching method can be executed by the terminal or the server, or by the terminal and the server working together.

[0091] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device uses the image matching method of this embodiment to provide related services and enhance the driving experience. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server can install the server-side software of the image matching system, which can interact with the terminal. For example, the server can install corresponding software, which can be an application implementing the image matching method, but is not limited to the above forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus), or network communication methods, and this embodiment does not impose any restrictions.

[0092] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0093] The image matching method in the embodiments of the present invention is described below.

[0094] Figure 1 This is an optional flowchart of the image matching method provided in the embodiments of the present invention. Figure 1 The method may include, but is not limited to, steps S110 to S170. It is also understood that this embodiment... Figure 1 The order of steps S110 to S170 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0095] Step S110: Obtain the query image sequence.

[0096] In one embodiment, the query image sequence is a current environment image collected and sent by the robot. The query image sequence includes at least one time-series-based query image, and each query image corresponds to an image feature. For example, the query image is an image of the surrounding environment sampled over a continuous period of time. There may be differences between different query images. For example, multiple images collected over a preset time period can be used as query images. For instance, 10 query images collected over 10 seconds at one-second intervals can form a query image sequence. The query images may or may not have differences. If the robot is stationary, there will be no differences between the query images in the query image sequence. Using the query image sequence method can further reflect the multidimensional information of the current surrounding environment.

[0097] Step S120: Input each query image into the image feature extraction module for feature extraction to obtain an image feature sequence.

[0098] In one embodiment, the image feature extraction module includes at least one convolutional unit, each of which includes at least one convolutional layer, an activation layer, and a pooling layer connected in sequence.

[0099] Reference Figure 2 This is a schematic diagram of the image feature extraction module.

[0100] In this embodiment, the image feature extraction module 100 includes five convolutional units connected in sequence, namely the first convolutional unit 110, the second convolutional unit 120, the third convolutional unit 130, the fourth convolutional unit 140, and the fifth convolutional unit 150.

[0101] The first convolutional unit 110 includes, in sequence: a first convolutional layer 111 (e.g., a 3*3 kernel and 64 channels), a second convolutional layer 112 (e.g., a 3*3 kernel and 64 channels), and a first pooling layer 113 (e.g., a 2*2 filter and a stride of 2), which is activated using a ReLU activation layer.

[0102] The second convolutional unit 120 includes, in sequence: a third convolutional layer 121 (e.g., a 3*3 kernel and 128 channels), a fourth convolutional layer 122 (e.g., a 3*3 kernel and 128 channels), and a second pooling layer 123 (e.g., a 2*2 filter with a stride of 2), which is activated using a ReLU activation layer.

[0103] The third convolutional unit 130 includes, in sequence: a fifth convolutional layer 131 (e.g., a 3*3 kernel with 256 channels), a sixth convolutional layer 132 (e.g., a 3*3 kernel with 256 channels), a seventh convolutional layer 133 (e.g., a 3*3 kernel with 256 channels), and a third pooling layer 134 (e.g., a 2*2 filter with a stride of 2), which is activated using a ReLU activation layer.

[0104] The fourth convolutional unit 140 includes, in sequence: an eighth convolutional layer 141 (e.g., a 3*3 kernel and 512 channels), a ninth convolutional layer 142 (e.g., a 3*3 kernel and 512 channels), a tenth convolutional layer 143 (e.g., a 3*3 kernel and 512 channels), and a fourth pooling layer 144 (e.g., a 2*2 filter with a stride of 2), which is activated using a ReLU activation layer.

[0105] The fifth convolutional unit 150 comprises, in sequence: an eleventh convolutional layer 151 (e.g., a 3x3 kernel with 512 channels), a twelfth convolutional layer 152 (e.g., a 3x3 kernel with 512 channels), a thirteenth convolutional layer 153 (e.g., a 3x3 kernel with 512 channels), and a fifth pooling layer 154 (e.g., a 2x2 filter with a stride of 2), activated using a ReLU activation layer. The fifth pooling layer is an adaptive max-pooling layer, capable of controlling the output parameter size based on the input parameters.

[0106] As can be seen from the above, the image feature extraction module of this application embodiment is more suitable for extracting scene image features. It is a deep learning network belonging to classification, which uses convolutional layers to extract features. Furthermore, the last pooling layer is an adaptive max-pooling layer, which can reduce the network training parameters and obtain smaller feature shapes to accelerate the model training process. In this application embodiment, each query image is input into the image feature extraction module 100 for feature extraction, resulting in an image feature sequence. Specifically, the query images are sequentially input into convolutional units for convolution, activation, and pooling operations to obtain an image feature sequence. This image feature sequence includes at least one image feature, with each query image corresponding to one image feature. That is, the number of image features corresponding to the number of query images forms the image feature sequence.

[0107] In the above embodiments, since the query image sequence includes at least one query image based on a time series, the image features of each query image have a temporal relationship, that is, the image feature sequence contains the image features of the query images at multiple times according to the temporal relationship.

[0108] Combination Figure 2In one embodiment, it is assumed that the size of the input query images is 224×224×3, and they are input into the image feature extraction module 100.

[0109] First, the first convolutional layer 111 and the second convolutional layer 112 of the first convolutional unit 110 perform two convolutions with a stride of 1 and padding of the same value. After the two convolutions, the image is activated by a ReLU activation layer, resulting in an output feature size of 224×224×64. In this embodiment, the first pooling layer 113 uses max pooling with a 2×2 filter and a stride of 2. Therefore, after pooling by the first pooling layer 113, the image size is halved, becoming 112×112×64.

[0110] Then, the third convolutional layer 121 and the fourth convolutional layer 122 of the second convolutional unit 120 undergo two convolutions, followed by activation by a ReLU activation layer, resulting in an output feature size of 112×112×128. In this embodiment, the second pooling layer 123 uses max pooling with a 2×2 filter and a stride of 2. Therefore, after pooling by the second pooling layer 123, the image size is halved, becoming 56×56×128.

[0111] The fifth convolutional layer 131, the sixth convolutional layer 132, and the seventh convolutional layer 133 of the third convolutional unit 130 undergo three convolutions, followed by activation by a ReLU activation layer, resulting in an output feature size of 56×56×256. In this embodiment, the third pooling layer 134 uses max pooling with a 2×2 filter and a stride of 2. Therefore, after pooling by the third pooling layer 134, the image size is halved, becoming 28×28×256.

[0112] The eighth, ninth, and tenth convolutional layers 141, 142, and 143 of the fourth convolutional unit 140 undergo three convolutions, followed by ReLU activation, resulting in an output feature size of 28×28×512. In this embodiment, the fourth pooling layer 144 uses max pooling with a 2×2 filter and a stride of 2. Therefore, after pooling by the fourth pooling layer 144, the image size is halved, becoming 14×14×512.

[0113] Finally, the eleventh convolutional layer 151, the twelfth convolutional layer 152, and the thirteenth convolutional layer 153 of the fifth convolutional unit 150 undergo three convolutions, followed by activation by a ReLU activation layer, resulting in an output feature size of 14×14×512. In this embodiment, the fifth pooling layer 154 uses adaptive max pooling, so after pooling by the fifth pooling layer 154, the size becomes 2×2×512.

[0114] As can be seen from the above, in this embodiment of the application, the image feature extraction module is used to extract features from each query image to obtain corresponding image features, and the image features are used to form an image feature sequence. The following steps process the image feature sequence.

[0115] Step S130: Input the image feature sequence into the residual feature processing module to extract residual features and obtain the residual fusion feature sequence.

[0116] In one embodiment, reference is made to Figure 3 The residual feature processing module 200 includes a residual feature extraction module 210 and a residual feature fusion module 220, wherein the image feature sequence first enters the residual feature extraction module 210, and then enters the residual feature fusion module 220 to obtain the residual fused feature sequence.

[0117] In one embodiment, reference is made to Figure 4 This is a flowchart illustrating a specific implementation of step S130 in one embodiment. In this embodiment, step S130, which involves inputting the image feature sequence into the residual feature processing module for residual feature extraction to obtain the residual fusion feature sequence, includes, but is not limited to, steps S131 to S132.

[0118] Step S131: Input the image feature sequence into the residual feature extraction module to extract residual features and obtain the residual feature sequence.

[0119] In one embodiment, compared to directly using feature values, residual features can combine prior knowledge of the image and use residual features to replace image features, thereby eliminating the impact of appearance changes on image matching similarity and improving search accuracy.

[0120] In one embodiment, reference is made to Figure 5 This is a flowchart illustrating a specific implementation of step S131 in one embodiment. In this embodiment, step S131, which involves inputting the image feature sequence into the residual feature extraction module to extract residual features and obtain the residual feature sequence, includes, but is not limited to, steps S1311 to S1315.

[0121] Step S1311: Obtain the first sliding window feature sequence of the image feature sequence within the first preset time window.

[0122] Step S1312: Obtain the second sliding window feature sequence of the image feature sequence within the second preset time window.

[0123] In one embodiment, since the image feature sequence contains image features of the query image at multiple time points obtained according to temporal relationships, it is assumed that the query image sequence is represented as: [q1,q2,...,q nThe query image contains n time points. For each query image, the image feature extraction module is used to extract image features, forming an image feature sequence, represented as: [x1, x2, ..., x...]. n ], x∈R D .

[0124] In one embodiment, the length of the first preset time window can be L1, and the length of the second preset time window can be L2. The length L of the first preset time window and the second preset time window is L = L1 + L2, where the length L represents the time span for calculating the residual. This embodiment does not limit the relationship between L1 and L2; L1 can be equal to L2, in which case L1 = L2 = L / 2.

[0125] In this embodiment, if the current time is t, the image feature at time t is x. t The first preset time window is represented as: [(t-L1),(t-1)], the second preset time window is represented as: [(t+1),(t+L2)], and the first sliding window feature sequence is represented as: [x t-L1 ,...,x t-1 The second sliding window feature sequence is represented as: [x] t+1 ,...,x t+L2 ].

[0126] Step S1313: Calculate the first residual feature based on the first sliding window feature sequence.

[0127] Step S1314: Calculate the second residual feature based on the second sliding window feature sequence.

[0128] In one embodiment, based on the first sliding window feature sequence and the image features x at the current time... t The first residual characteristic a1 is calculated and expressed as:

[0129]

[0130] Based on the second sliding window feature sequence and the current time image feature x t The second residual characteristic a2 is calculated and expressed as:

[0131]

[0132] Step S1315: Calculate the residual feature sequence based on the first residual feature and the second residual feature.

[0133] In one embodiment, the residual feature sequence a is calculated based on the first residual feature a1 and the second residual feature a2. t a t ∈R 2×D , is represented as:

[0134]

[0135] For example, in one embodiment, the image feature sequence is represented as:

[0136] [x1,x2,x3,x4,x5,x6,x7,x8,x9,x 10 ]

[0137] The first preset time window L1 is 2, the second preset time window L2 is 2, and when the current time t is 3, the image feature at time t is x3. Then the first preset time window is represented as [1,2], the second preset time window is represented as [4,5], the first sliding window feature sequence is represented as [x1,x2], and the second sliding window feature sequence is represented as [x4,x5].

[0138] The first residual feature is calculated based on the first sliding window feature sequence and the current time image feature x3, and is expressed as: x3-2*(x1+x2); the second residual feature is calculated based on the second sliding window feature sequence and the current time image feature x3, and is expressed as: 2*(x4+x5)-x3.

[0139] The residual feature sequence calculated based on the first and second residual features is expressed as follows:

[0140] a3=[(x3-2*(x1+x2)),(2*(x4+x5)-x3)]

[0141] As described above, the residual feature sequence combines information from image features centered at time t, with time windows of length 1 and 2, respectively, preserving the trend of image feature values ​​through residuals. This smooth residual feature avoids numerical shifts between the query image features and the image features in the database due to appearance changes, thereby improving the accuracy of subsequent similarity matching. Furthermore, in this embodiment, the residual feature sequence only preserves the trend of image feature values, avoiding direct use of image feature values ​​and thus preventing the impact of numerical shifts on search accuracy. In addition, the residual feature sequence is calculated based on image features before and after time t, making the values ​​of the residual feature sequence smoother and reducing errors caused by noise between query image sequences. Therefore, this embodiment uses the residual feature sequence for calculation to obtain more accurate similarity matching results.

[0142] Step S132: Input the residual feature sequence into the residual feature fusion module to perform residual feature fusion and obtain the residual fused feature sequence.

[0143] In one embodiment, the residual feature fusion module is an LSTM model. (Refer to...) Figure 6 A schematic diagram of the residual feature fusion module.

[0144] The residual feature fusion module includes an input gate, a forget gate, and an output gate. As shown in the figure, at time t, the residual feature fusion module has three inputs: the current input value s. t The output value h from the previous time step t-1 and the cell state c from the previous time step t-1 The residual feature fusion module has two components: the output value h at the current time. t and the current cell state c t .

[0145] Wherein, the input value s at the current time is t and the output value h from the previous time step t-1 The values ​​are concatenated, multiplied by the first weight matrix wi, and then converted into values ​​between 0 and 1 by the sigmoid activation function to obtain the first gated state value ft.

[0146] The input value s at the current moment t and the output value h from the previous time step t-1 The values ​​are concatenated, multiplied by the second weight matrix wf, and then converted into values ​​between 0 and 1 using the sigmoid activation function to obtain the second gated state value it.

[0147] The input value s at the current moment t and the output value h from the previous time step t-1 The values ​​are concatenated, multiplied by the third weight matrix wo, and then converted into values ​​between 0 and 1 by the sigmoid activation function to obtain the third gated state value ot.

[0148] Next, the input value s at the current moment. t and the output value h from the previous time step t-1 The data is concatenated and then converted to a value between -1 and 1 using the tanh activation function to obtain new information mt.

[0149] Finally, perform the following calculations in sequence:

[0150] 1) Set the cell state c from the previous time step. t-1 Perform Hadamard calculation on the first gated state value ft, and multiply the corresponding elements in the two matrices to obtain the first intermediate state value k1;

[0151] 2) Perform Hadamard calculation on the second gated state value it and the new information mt, and multiply the corresponding elements in the two matrices to obtain the second intermediate state value k2;

[0152] 3) Add the first intermediate state value k1 and the second intermediate state value k2 to obtain the current cell state c. t ;

[0153] 4) Set the current cell state c t After being transformed into a value between -1 and 1 by the tanh activation function, the third intermediate state value k3 is obtained; the third intermediate state value k3 is the current cell state c. t .

[0154] 5) Perform Hadamard calculation on the third intermediate state value k3 and the third gated state value ot, and multiply the corresponding elements in the two matrices to obtain the output value h at the current time. t .

[0155] In one embodiment, reference is made to Figure 7 This is a flowchart illustrating a specific implementation of step S132 in one embodiment. In this embodiment, step S132, in which the residual feature sequence is input into the residual feature fusion module for residual feature fusion to obtain the residual fused feature sequence, includes, but is not limited to, steps S1321 to S1322.

[0156] Step S1321: Input the first residual feature into the residual feature fusion module to obtain the first fusion output feature.

[0157] Step S1322: Input the first fusion output feature and the second residual feature into the residual feature fusion module to obtain the residual fusion feature sequence.

[0158] Reference Figure 8 In one embodiment, a residual feature fusion module is used to learn the temporal relationship between residual features. Since the residual feature sequence a... t It consists of a first residual feature a1 and a second residual feature a2. Therefore, the first residual feature a1 is first input into the residual feature fusion module for calculation to obtain the first fusion output feature. Then, the first fusion output feature and the second residual feature a2 are input together into the residual feature fusion module for calculation to obtain the residual fusion feature sequence.

[0159] Reference Figure 6 At time t, the output value h from the previous time step is... t-1 and the cell state c at the previous time step t-1 Since the numbers are set to random numbers that follow a normal distribution, the input value of the residual feature fusion module is: the input value s at time t. t (First residual feature a1), output value h from the previous time step t-1 and the cell state c from the previous time step t-1 The output value obtained is: the output value h at time t. t and the cell state c at time t t At time t+1, the input value of the residual feature fusion module is: the input value s at time t+1. t (Second residual feature a2), output value h at time t tand the cell state c at time t t The output value obtained is: the output value h at time t+1. t+1 and the cell state c at time t+1 t+1 At this point, the output value h at time t+1 is... t+1 This is the residual fusion feature sequence of this embodiment.

[0160] As can be seen from the above, the residual feature fusion module obtains the residual fusion feature sequence based on the first residual feature and the second residual feature of the residual feature sequence.

[0161] In one embodiment, the residual feature processing module needs to be pre-trained, referring to... Figure 9 This is a flowchart illustrating a specific implementation of a pre-trained residual feature processing module. In this embodiment, the steps of pre-training the residual feature processing module include, but are not limited to, steps S910 to S960.

[0162] Step S910: Obtain the training sample set.

[0163] In one embodiment, the training sample set includes: training image samples and fused labels, wherein the fused labels include a first label and a second label. For example, the first label may be image features of positive samples, and the second label may be image features of negative samples. Positive samples may be images located in the same place as training image samples but with different appearance variations, and negative samples may be images located in different places than training image samples with different appearance variations. In one embodiment, positive and negative samples can be selected based on GPS positioning information provided in a database using filtering rules.

[0164] Step S920: Input the training image samples into the residual feature extraction module and the residual feature fusion module in sequence to obtain the residual fusion prediction sequence.

[0165] In one embodiment, the residual feature processing module includes the residual feature extraction module and the residual feature fusion module described above, and calculates the residual fusion prediction sequence of the training image samples in the manner described above.

[0166] Step S930: Calculate the first error value between the residual fusion prediction sequence and the first label.

[0167] Step S940: Calculate the second error value between the residual fusion prediction sequence and the second label.

[0168] In one embodiment, the residual fusion prediction sequence is denoted as xa, the first label is denoted as xpos, and the second label is denoted as xneg. Then, the first error value is denoted as:

[0169] ||xa-xpos||2

[0170] The second error value is expressed as:

[0171] ||xa-xneg||2

[0172] Step S950: Calculate the loss value based on the first error value and the second error value.

[0173] In one embodiment, the loss value is expressed as:

[0174] loss=max(||xa-xpos||2-||xa-xneg||2+m)

[0175] Here, loss represents the loss value, and m represents the hyperparameter, which is used to adjust the distance between positive and negative samples in the feature space. It can be obtained based on the actual testing process or experience.

[0176] Step S960: Adjust the model parameters of the residual feature processing module according to the loss value.

[0177] It is understandable that the weights of the residual feature extraction module and the residual feature fusion module are adjusted according to the loss value until the loss value meets the iteration condition. The iteration condition can be to meet a certain number of iterations, or the loss value can be less than a preset reference value, etc. This embodiment does not impose specific limitations on the iteration condition, and it can be selected according to the actual situation.

[0178] Step S140: Calculate the similarity between the residual fused feature sequence and at least one matching feature sequence to obtain the similarity value between the fused feature sequence and each matching feature sequence.

[0179] In one embodiment, the database stores multiple sets of matching image sequences, each containing matching images corresponding to the number of query images in the query image sequence. Features are extracted from each matching image beforehand using an image feature extraction module to obtain matching features for each matching image, thereby obtaining a matching feature sequence for each matching image sequence. It is understood that the feature value dimension in the matching feature sequence is consistent with the feature value dimension in the residual fusion feature sequence.

[0180] In one embodiment, the residual fusion feature sequence is matched one by one with all pre-stored matching feature sequences in the database. Specifically, the cosine similarity between the two vectors is calculated, and the cosine similarity value is used as the matching information. The closer the cosine similarity value is to 1, the more likely it is that the two images are located at the same location. If it is close to -1, it means that the two images indicate different locations.

[0181] Step S150: Select multiple matching feature sequences based on similarity values ​​to obtain candidate image sequences.

[0182] In one embodiment, the similarity value is the aforementioned cosine similarity value. First, all cosine similarity values ​​are sorted, and the matching feature sequences corresponding to the top-ranked cosine similarity values ​​are selected as candidate image sequences according to a preset number of candidates. For example, if there are F matching image sequences in the database, the top N matching image sequences are selected as candidate image sequences.

[0183] Step S160: Calculate the similarity matrix based on the image feature sequence and the candidate image sequence.

[0184] In one embodiment, reference is made to Figure 10 This is a flowchart illustrating a specific implementation of step S160 in one embodiment. In this embodiment, step S160, which calculates the similarity matrix based on the image feature sequence and the candidate image sequence, includes, but is not limited to, steps S161 to S162.

[0185] Step S161: Calculate the matching similarity matrix between the image feature sequence and each matching feature sequence.

[0186] Reference Figure 11 The diagram illustrates the similarity matrix. Assume the query image sequence contains k query images, each corresponding to an image feature. These k query images' features form an image feature sequence. Correspondingly, N matching image sequences are selected from the database's matching image sequences as candidate image sequences. This means each candidate image sequence contains N matching image sequences, each matching image sequence contains k matching images, each matching image corresponds to a matching feature, and each matching feature sequence contains k matching features.

[0187] In one embodiment, calculating the matching similarity matrix between the image feature sequence and each matching feature sequence specifically includes: sequentially matching each feature value in the image feature sequence with each feature value in the matching feature sequence to obtain the row vector of the matching similarity matrix, and then constructing the matching similarity matrix based on the row vector.

[0188] In one embodiment, each feature value in the image feature sequence is calculated sequentially; that is, the vector value is calculated one by one between the image features and the matching features in the matching feature sequence. (Refer to...) Figure 11 For example, if k=4, then the image feature sequence contains k image features. Correspondingly, each of the N matching image sequences contains k matching features. First, the first image feature in the image feature sequence is multiplied by the k matching features in the first matching image sequence to obtain the first row vector. Then, the second image feature in the image feature sequence is multiplied by the k matching features in the first matching image sequence to obtain the second row vector, and so on, until the fourth row vector is obtained.

[0189] In one embodiment, reference is made to Figure 11 The first row vector, the second row vector, and so on up to the fourth row vector obtained above are concatenated row by row to obtain the matching similarity matrix of the first matching image sequence, where the matching similarity matrix is ​​a 4-row, 4-column matrix.

[0190] The matching similarity matrices of N matching image sequences are obtained in sequence. When the image feature sequence contains k image features, the size of the matching similarity matrix is ​​[k, k].

[0191] Step S162: Combine the matching similarity matrices to obtain the similarity matrix.

[0192] Reference Figure 11 In one embodiment, the matching similarity matrices are combined into a similarity matrix, wherein the similarity matrix is ​​N [k,k] matrices.

[0193] Step S170: Select the matching image sequence corresponding to the matching feature sequence based on the similarity matrix to obtain the target image.

[0194] In one embodiment, reference is made to Figure 12 This is a flowchart illustrating a specific implementation of step S170 in one embodiment. In this embodiment, step S170, which calculates the matching similarity matrix between the image feature sequence and each matching feature sequence, includes, but is not limited to, steps S171 to S174.

[0195] Step S171: Calculate the mean of the values ​​on the diagonal of each matching similarity matrix.

[0196] In one embodiment, reference is made to Figure 11 For each matching similarity matrix, the mean of the values ​​on the diagonal from left to right is calculated.

[0197] Step S172: Select the matching feature sequence corresponding to the matching similarity matrix with the largest mean.

[0198] In one embodiment, the matching similarity matrix with the largest mean is used as the target matrix, and the matching feature sequence of the target matrix is ​​selected.

[0199] Step S173: Obtain the matching image sequence corresponding to the matching feature sequence as the target image sequence.

[0200] In one embodiment, the matching feature sequence corresponds one-to-one with the matching image sequence, so the matching image sequence corresponding to the matching feature sequence can be obtained as the target image sequence.

[0201] Step S174: Obtain the target image based on the target image sequence.

[0202] In one embodiment, reference is made to Figure 11The matching image and the query image are selected from the matching similarity matrix of the matching feature sequence of the target image sequence, and the matching image is used as the target image corresponding to the query image.

[0203] In one embodiment, assuming the query image sequence is [q1,q2,…,qn], the target image sequence obtained after the above steps is [r1,r2,…,rn]. Generally, matching means that qi and ri need to be matched one-to-one. However, in reality, continuous images over a period of time do not change much between frames; for example, the content of images q1 and r2 may still be the same location. Therefore, this embodiment appropriately increases the precision of the matching width to improve matching efficiency. For a single frame of query image qi and target image rj, when i≠j, if |ij|<4, then the match is considered successful.

[0204] This application embodiment extracts residual features from the image feature sequence using a residual feature processing module, and fuses these residual features to obtain a residual fused feature sequence. Combining prior knowledge of the image, the residual features are used to replace image features, thereby eliminating the influence of appearance changes on image matching similarity and improving search accuracy. Furthermore, through multiple sets of experiments comparing the image with location recognition methods in related technologies, and verification on several commonly used datasets, it was found that the image matching method of this application embodiment significantly improves the accuracy of similarity matching.

[0205] Reference Figure 13 This is a flowchart of an image matching process according to an embodiment of this application.

[0206] First, a query image sequence is input, which includes multiple time-series-based query images. Then, the image feature extraction module extracts features from each query image to obtain corresponding image features, and these image features are combined into an image feature sequence.

[0207] Then it is processed in two branches.

[0208] The first branch inputs the image feature sequence into the residual feature extraction module for residual feature extraction, resulting in a residual feature sequence. Compared to directly using feature values, residual features can combine prior knowledge of the image, using residual features to replace image features to eliminate the influence of appearance changes on image matching similarity and improve search accuracy.

[0209] The residual feature sequence is then input into the residual feature fusion module for residual feature fusion, learning the temporal relationship between residual features to obtain the residual fused feature sequence.

[0210] The image feature extraction module is used to extract features from each matching image in the database in advance to obtain the matching features of each matching image, and then the matching feature sequence of each matching image sequence is obtained.

[0211] Then, the similarity between the residual fused feature sequence and at least one matching feature sequence is calculated to obtain the similarity value between the fused feature sequence and each matching feature sequence.

[0212] Then, based on the similarity value, multiple matching feature sequences are selected to obtain N candidate image sequences.

[0213] The second branch calculates the similarity matrix based on the image feature sequence and the candidate image sequence. First, each feature value in the image feature sequence is matched one-to-one with each feature value in the matching feature sequence, resulting in row vectors of the matching similarity matrix. Then, the matching similarity matrix is ​​constructed based on these row vectors. This process is repeated for N matching image sequences. When the image feature sequence contains k image features, the size of the matching similarity matrix is ​​[k, k]. Finally, the matching similarity matrices are combined to obtain the final similarity matrix, which consists of N [k, k] matrices.

[0214] Finally, the target image is obtained by selecting the matching image sequence corresponding to the matching feature sequence based on the similarity matrix. Let's assume it's the last matching image sequence in the diagram. Specifically: calculate the mean of the values ​​on the diagonal of each matching similarity matrix (shown by the black lines in the diagram), select the matching feature sequence corresponding to the matching similarity matrix with the largest mean, obtain the matching image sequence corresponding to the matching feature sequence as the target image sequence, and then obtain the target image based on the target image sequence.

[0215] The technical solution provided by this invention involves extracting features from each query image in the obtained query image sequence to obtain an image feature sequence. Then, the image feature sequence is input into a residual feature processing module for residual feature extraction. The residual features are then fused to obtain a residual fused feature sequence. The residual fused feature sequence is then compared with at least one matching feature sequence to obtain a similarity value between the fused feature sequence and each matching feature sequence. Multiple matching feature sequences are then selected based on the similarity values ​​to obtain a candidate image sequence. Next, a similarity matrix is ​​calculated based on the image feature sequence and the candidate image sequence. Finally, the matching image sequence corresponding to the matching feature sequence is selected based on the similarity matrix to obtain the target image.

[0216] This invention also provides a positioning method applicable to intelligent robot devices, such as drones and robot dogs.

[0217] In one embodiment, the localization method includes: acquiring multiple environmental images as a query image sequence, executing an image matching method as described in any of the above embodiments, obtaining a target image based on the query image sequence, and finally locating the target object based on the target image and a preset image-location mapping relationship.

[0218] Reference Figure 14 This is a schematic diagram of an application scenario for the positioning method one in this application embodiment.

[0219] The intelligent robot 140 in the picture includes:

[0220] Image acquisition device 1410 is used to acquire multiple environmental images as a query image sequence. It is understood that image acquisition device 1410 can be one or more cameras.

[0221] The processing unit 1420 is used to execute the image matching method as described in the above embodiment, obtain the target image based on the query image sequence acquired by the image acquisition device 1410, and locate the target image to obtain positioning information.

[0222] Furthermore, the processing unit 1420 is also used to generate further driving information based on the task being performed and the positioning information, and send it to the driving device 1430. The driving unit 1430 uses the driving information to drive the robot body 1440 of the intelligent robot 140 to move to a designated position, or to perform a designated action, in order to complete the task being performed.

[0223] In one embodiment, illustrated by dashed lines in the figure, the intelligent robot 140 can also use the communication unit 1450 to send the query image sequence to the external server 1460. The external server 1460 executes the image matching method as described in the above embodiment, obtains the target image based on the query image sequence, and performs positioning based on the target image to obtain positioning information or further driving information. Then, it uses the communication unit 1450 to receive the driving information and sends it to the driving unit 1430 to drive the robot body 1440 of the intelligent robot 140 to move to a designated position or perform a designated action to complete the task.

[0224] To facilitate understanding of the embodiments of this application, the application scenarios of the embodiments of this application will be briefly introduced below with examples of specific application scenarios.

[0225] In a specific application scenario, intelligent robots assist people in completing production tasks at work sites. The robots utilize their own positioning systems to move in real-time to the locations required to complete the tasks. While the robot is running, it inputs images of its surrounding environment as query image sequences into a visual position recognition device (such as the processing unit mentioned above). The visual position recognition device extracts features containing scene location information from the images. Considering the impact of appearance changes such as seasons, lighting, and weather on image content, a residual feature processing module extracts residual features from the image feature sequence and fuses these residual features to obtain a residual fusion feature sequence. Combined with prior knowledge of the image, the residual features are used to replace the image features, eliminating the impact of appearance changes on image matching similarity. The obtained residual fusion feature sequence is used to search for nearest neighbor images in the database. The geographical information of the nearest neighbor images is used as the current geographical information of the robot, thereby correcting the robot's offset position on the map, maintaining the consistency between the robot's physical position and its map position, and ultimately guiding the intelligent robot to complete the relevant production tasks.

[0226] The localization method of this application embodiment utilizes the image matching method of the above embodiment to extract residual features from the image feature sequence through the residual feature processing module, and fuses the residual features to obtain a residual fused feature sequence. Combined with prior knowledge of the image, the residual features are used to replace the image features in order to eliminate the influence of appearance changes on image matching similarity, improve matching accuracy, and thus improve localization accuracy.

[0227] This invention also provides an image matching apparatus that can implement the above-described image matching method, see reference. Figure 15 The device includes:

[0228] The acquisition unit 1510 is used to acquire a query image sequence, which includes at least one query image based on a time series.

[0229] The image feature extraction unit 1520 is used to input each query image into the image feature extraction module for feature extraction to obtain an image feature sequence.

[0230] The residual feature extraction unit 1530 is used to input the image feature sequence into the residual feature processing module for residual feature extraction to obtain the residual fusion feature sequence.

[0231] The first similarity calculation unit 1540 is used to calculate the similarity between the residual fused feature sequence and at least one matching feature sequence to obtain the similarity value between the fused feature sequence and each matching feature sequence.

[0232] The candidate image sequence selection unit 1550 is used to select multiple matching feature sequences based on similarity values ​​to obtain candidate image sequences.

[0233] The second similarity calculation unit 1560 is used to calculate the similarity matrix based on the image feature sequence and the candidate image sequence.

[0234] The target image sequence selection unit 1570 is used to select the matching image sequence corresponding to the matching feature sequence according to the similarity matrix to obtain the target image.

[0235] The specific implementation of the image matching device in this embodiment is basically the same as the specific implementation of the image matching method described above, and will not be repeated here.

[0236] This invention also provides an electronic device, comprising:

[0237] At least one memory;

[0238] At least one processor;

[0239] At least one program;

[0240] The program is stored in a memory, and the processor executes the at least one program to implement the image matching method described above in this invention. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0241] Please see Figure 16 , Figure 16 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0242] The processor 1601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0243] The memory 1602 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1602 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1602 and is called and executed by the processor 1601 to execute the image matching method of the embodiments of this invention.

[0244] The input / output interface 1603 is used to implement information input and output;

[0245] The communication interface 1604 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0246] Bus 1605 transmits information between various components of the device (e.g., processor 1601, memory 1602, input / output interface 1603, and communication interface 1604);

[0247] The processor 1601, memory 1602, input / output interface 1603 and communication interface 1604 are connected to each other within the device via bus 1605.

[0248] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described image matching method.

[0249] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0250] The image matching method, image matching device, electronic device, and storage medium proposed in this invention extract features from each query image in the acquired query image sequence to obtain an image feature sequence. This image feature sequence is then input into a residual feature processing module for residual feature extraction. The residual features are fused to obtain a residual fused feature sequence. The residual fused feature sequence is then compared with at least one matching feature sequence to calculate a similarity value. Multiple matching feature sequences are selected based on the similarity values ​​to obtain a candidate image sequence. A similarity matrix is ​​then calculated based on the image feature sequence and the candidate image sequences. Finally, the matching image sequence corresponding to the matching feature sequence is selected based on the similarity matrix to obtain the target image. This embodiment of the application extracts residual features from the image feature sequence using a residual feature processing module and fuses these residual features to obtain a residual fused feature sequence. By combining prior knowledge of the image and using residual features to replace image features, the influence of appearance changes on image matching similarity is eliminated, thereby improving search accuracy.

[0251] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0252] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0253] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0254] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0255] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0256] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0258] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0259] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0260] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image matching method, characterized in that, include: Obtain a query image sequence, the query image sequence including at least one query image based on a time series; Each query image is input into the image feature extraction module for feature extraction to obtain an image feature sequence; The image feature sequence is input into the residual feature processing module for residual feature extraction to obtain a residual fusion feature sequence. The residual feature processing module includes a residual feature extraction module and a residual feature fusion module. The similarity between the residual fusion feature sequence and at least one matching feature sequence is calculated to obtain the similarity value between the residual fusion feature sequence and each of the matching feature sequences. Multiple matching feature sequences are selected based on the similarity values ​​to obtain candidate image sequences; Calculate a similarity matrix based on the image feature sequence and the candidate image sequence; The target image is obtained by selecting the matching image sequence corresponding to the matching feature sequence based on the similarity matrix; The step of inputting the image feature sequence into the residual feature processing module for residual feature extraction to obtain a residual fusion feature sequence includes: inputting the image feature sequence into the residual feature extraction module for residual feature extraction, obtaining a first sliding window feature sequence of the image feature sequence within a first preset time window, obtaining a second sliding window feature sequence of the image feature sequence within a second preset time window, calculating a first residual feature based on the first sliding window feature sequence, calculating a second residual feature based on the second sliding window feature sequence, calculating the residual feature sequence based on the first residual feature and the second residual feature, inputting the first residual feature into the residual feature fusion module to obtain a first fusion output feature, and inputting the first fusion output feature and the second residual feature into the residual feature fusion module to obtain the residual fusion feature sequence.

2. The image matching method according to claim 1, characterized in that, The image feature extraction module includes at least one convolutional unit, and each convolutional unit includes at least one convolutional layer, an activation layer and a pooling layer connected in sequence. The step of inputting each query image into the image feature extraction module for feature extraction to obtain an image feature sequence includes: sequentially inputting the query images into the convolution unit for convolution, activation and pooling operations to obtain an image feature sequence; the image feature sequence includes at least one image feature, and each query image corresponds to one image feature.

3. The image matching method according to claim 1, characterized in that, Before inputting the image feature sequence into the residual feature processing module for residual feature extraction to obtain the residual fused feature sequence, the method further includes: training the residual feature processing module, the training process including: Obtain a training sample set, which includes: training image samples and fusion labels, wherein the fusion labels include a first label and a second label; The training image samples are sequentially input into the residual feature extraction module and the residual feature fusion module to obtain the residual fusion prediction sequence; Calculate the first error value between the residual fusion prediction sequence and the first label; Calculate the second error value between the residual fusion prediction sequence and the second label; The loss value is calculated based on the first error value and the second error value; The model parameters of the residual feature processing module are adjusted based on the loss value.

4. The image matching method according to claim 1, characterized in that, The candidate image sequence includes at least one matching feature sequence, which is obtained by inputting the matching image into the image feature extraction module for feature extraction. The step of calculating the similarity matrix based on the image feature sequence and the candidate image sequence includes: Calculate the matching similarity matrix between the image feature sequence and each of the matching feature sequences; The similarity matrix is ​​obtained by combining the matching similarity matrix.

5. The image matching method according to claim 4, characterized in that, The calculation of the matching similarity matrix between the image feature sequence and each of the matching feature sequences includes: Each feature value in the image feature sequence is matched one by one with each feature value in the matching feature sequence to obtain the row vector of the matching similarity matrix; The matching similarity matrix is ​​constructed based on the row vectors.

6. The image matching method according to claim 4, characterized in that, The step of selecting the matching image sequence corresponding to the matching feature sequence based on the similarity matrix to obtain the target image includes: Calculate the mean of the values ​​on the diagonal of each of the matching similarity matrices; Select the matching feature sequence corresponding to the matching similarity matrix with the largest mean; Obtain the matching image sequence corresponding to the matching feature sequence as the target image sequence; The target image is obtained from the target image sequence.

7. A positioning method, characterized in that, include: Obtain multiple environmental images as a query image sequence; The image matching method as described in any one of claims 1 to 6 is executed to obtain the target image based on the query image sequence; The target object is located based on the target image and the preset image-position mapping relationship.

8. An image matching device, characterized in that, include: An acquisition unit is used to acquire a query image sequence, the query image sequence including at least one query image based on a time series. An image feature extraction unit is used to input each query image into the image feature extraction module for feature extraction to obtain an image feature sequence; The residual feature extraction unit is used to input the image feature sequence into the residual feature processing module for residual feature extraction to obtain a residual fusion feature sequence. The residual feature processing module includes a residual feature extraction module and a residual feature fusion module. The first similarity calculation unit is used to calculate the similarity between the residual fusion feature sequence and at least one matching feature sequence to obtain the similarity value between the residual fusion feature sequence and each of the matching feature sequences. A candidate image sequence selection unit is used to select multiple matching feature sequences based on the similarity value to obtain a candidate image sequence; The second similarity calculation unit is used to calculate a similarity matrix based on the image feature sequence and the candidate image sequence; The target image sequence selection unit is used to select a matching image sequence corresponding to the matching feature sequence according to the similarity matrix to obtain the target image; The step of inputting the image feature sequence into the residual feature processing module for residual feature extraction to obtain a residual fusion feature sequence includes: inputting the image feature sequence into the residual feature extraction module for residual feature extraction, obtaining a first sliding window feature sequence of the image feature sequence within a first preset time window, obtaining a second sliding window feature sequence of the image feature sequence within a second preset time window, calculating a first residual feature based on the first sliding window feature sequence, calculating a second residual feature based on the second sliding window feature sequence, calculating the residual feature sequence based on the first residual feature and the second residual feature, inputting the first residual feature into the residual feature fusion module to obtain a first fusion output feature, and inputting the first fusion output feature and the second residual feature into the residual feature fusion module to obtain the residual fusion feature sequence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the image matching method according to any one of claims 1 to 6, or the positioning method according to claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image matching method of any one of claims 1 to 6, or the positioning method of claim 7.

Images