Multispectral image registration methods, apparatus, computer equipment, and storage media

Abstract

This application relates to a multispectral image registration method, apparatus, computer device, computer-readable storage medium, and computer program product. The method includes: acquiring at least two original spectral images of a target object to be registered; locating the target object in each original spectral image to obtain target regions of the target object in each original spectral image; extracting features from each target region to obtain feature information corresponding to each target region; performing feature matching on each feature information to obtain a feature matching matrix; determining the transformation relationship between each original spectral image based on the feature matching matrix; and performing a projection transformation on each original spectral image based on the transformation relationship to obtain a registered spectral image. This method can effectively improve the registration accuracy of spectral images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multispectral image registration method, target recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Insulators or bushings in power systems can become contaminated and age over time, leading to grid failures. Multispectral imaging technology can be used to quickly, accurately, non-contactly, and non-destructively monitor the condition of insulators and bushings, thus preventing them from threatening the stable operation of the power grid.

[0003] Due to the trade-off between spatial and spectral resolution, monocular devices inevitably reduce spatial resolution while increasing spectral resolution, failing to meet the high-precision requirements of line inspection. In contrast, multi-view devices can reduce the spectral resolution of a single channel by increasing the number of lenses, ensuring high spatial resolution of the acquired images. Since different lenses capture the same target, even when two lenses are placed side-by-side, there is still a significant optical parallax. Furthermore, because the non-coaxial spectral images of the sheath typically contain a large amount of complex, non-coplanar background interference, direct image registration leads to the extraction and matching of numerous features outside the sheath area, making accurate registration of the target area impossible. This results in low registration accuracy, which in turn reduces the recognition accuracy of the sheath target area. Summary of the Invention

[0004] Therefore, it is necessary to provide a multispectral image registration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve registration accuracy in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a multispectral image registration method, the method comprising:

[0006] At least two original spectral images of the target object to be registered are acquired, and the target object in each of the original spectral images is located to obtain the target region of the target object in each of the original spectral images;

[0007] Feature extraction is performed on each of the target regions to obtain the feature information corresponding to each target region;

[0008] Perform feature matching on each of the aforementioned feature information to obtain a feature matching matrix;

[0009] Based on the feature matching matrix, the transformation relationship between each of the original spectral images is determined, and the original spectral images are projected and transformed based on the transformation relationship to obtain the registered spectral images.

[0010] In one embodiment, locating the target object in each of the original spectral images to obtain the target region of the target object in each of the original spectral images includes:

[0011] For each original spectral image, the target object in the original spectral image is identified by the target localization module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target region of the target object in each original spectral image.

[0012] In one embodiment, the feature information includes multiple feature points on the original spectral image that characterize the position of the target object in the original spectral image, and feature descriptors corresponding one-to-one with each feature point;

[0013] The step of extracting features from each of the target regions to obtain feature information corresponding to each target region includes:

[0014] For each target region, the target region is encoded by the feature extraction module of the trained image registration model to obtain the feature image of the target region;

[0015] The feature image is decoded to obtain the probability that each pixel in the feature image is a candidate point. The candidate points are then filtered based on the probability to obtain the feature points of the original spectral image.

[0016] The feature image is descriptor decoded to obtain the feature descriptor corresponding to the feature point.

[0017] In one embodiment, the step of performing feature matching on each of the feature information to obtain a feature matching matrix includes:

[0018] For any two target regions, the feature points and feature descriptors of the image registration model are used to determine the registration descriptors corresponding to each feature point in the two target regions.

[0019] Through the feature matching module, based on the registration descriptors corresponding to each feature point in the two target regions, the registration score of each registration descriptor in one of the two target regions and each registration descriptor in the other target region is determined, and a registration score matrix is ​​obtained.

[0020] The feature matching module determines the matching probability between the feature points corresponding to each registration descriptor in one of the two target regions and the feature points of each registration descriptor in the other target region, based on the registration scoring matrix, thereby obtaining the feature matching matrix.

[0021] In one embodiment, determining the registration descriptor corresponding to each feature point in any two target regions using the feature matching module of the image registration model for the feature points and feature descriptors of any two target regions includes:

[0022] For any two target regions, the feature points and feature descriptors of the image registration model are encoded into a first feature vector by the feature matching module of one of the target regions, and into a second feature vector by the feature points and feature descriptors of the other target region.

[0023] Based on the self-attention mechanism, the first feature vectors and the second feature vectors are fused and updated accordingly to obtain the corresponding first intermediate vector and second intermediate vector.

[0024] Based on the cross-attention mechanism, each first intermediate vector is fused with each second intermediate vector to update the first intermediate vector, resulting in a first update vector. Based on the first update vector, the registration descriptor of the feature points corresponding to each first feature vector is obtained. Based on the cross-sub-attention mechanism, each second intermediate vector is fused with each first intermediate vector to update the second intermediate vector, resulting in a second update vector. Based on the second update vector, the registration descriptor of the feature points corresponding to each second feature vector is obtained.

[0025] In one embodiment, before performing feature matching on each of the feature information to obtain a feature matching matrix, the method further includes:

[0026] Edge detection is performed on the target region to determine the edge regions of the target region, and the feature information corresponding to the edge regions is removed;

[0027] The step of performing feature matching on each of the aforementioned feature information to obtain a feature matching matrix includes:

[0028] Feature matching is performed on the feature information after removing the edge regions to obtain a feature matching matrix.

[0029] In one embodiment, the method further includes:

[0030] The registered spectral image is input into the target recognition model to identify the target object, and the recognition result of the target object is output.

[0031] Secondly, this application also provides a multispectral image registration device, the device comprising:

[0032] The target object localization module is used to acquire at least two original spectral images to be registered for the target object, and to locate the target object in each of the original spectral images to obtain the target region of the target object in each of the original spectral images;

[0033] The feature information extraction module is used to extract features from each of the target regions to obtain feature information corresponding to each of the target regions.

[0034] The feature information matching module is used to perform feature matching on each of the feature information to obtain a feature matching matrix;

[0035] The image registration module is used to determine the transformation relationship between each of the original spectral images based on the feature matching matrix, and to perform projection transformation on each of the original spectral images based on the transformation relationship to obtain the registered spectral images.

[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0039] The aforementioned multispectral image registration method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire at least two original spectral images of the target object to be registered, locate the target object in each original spectral image, and obtain the target region of the target object in the corresponding original spectral image. This process can remove background information from the original spectral images, enabling the location of the target object in a non-coaxial spectral image with a complex background. This allows subsequent registration processes to focus on the target region, improving registration accuracy and reducing the computational load, thus increasing registration efficiency. Subsequently, feature extraction is performed on the target objects in each target region to obtain the corresponding feature information. Feature matching is then performed on this feature information to obtain a feature matching matrix. Based on the feature matching matrix, the transformation relationship between the original spectral images is determined, and a projection transformation is performed on the original spectral images based on this transformation relationship to obtain the registered spectral image. This effectively extracts a large number of effective feature points of the target region in the original spectral images to be registered, improving the quality of feature point extraction. Furthermore, the registration of the spectral images based on the correspondence between feature points in different spectral images helps improve registration accuracy. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is an application environment diagram of a multispectral image registration method in one embodiment;

[0042] Figure 2 This is a flowchart illustrating a multispectral image registration method in one embodiment;

[0043] Figure 3 This is a schematic diagram of the feature extraction process for each target region in one embodiment;

[0044] Figure 4 This is a schematic diagram of the feature matching process for each feature information in one embodiment;

[0045] Figure 5 This is a schematic diagram of the process for determining a registration descriptor in one embodiment;

[0046] Figure 6 This is a flowchart illustrating the registration process between images of a first target region and images of a second target region in an application example.

[0047] Figure 7 This is a structural block diagram of a multispectral image registration device in one embodiment;

[0048] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] The multispectral image registration method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 locates the target object in the original spectral image of the target object, obtaining the position of the target object in the corresponding original spectral image to obtain the target region. Subsequently, server 104 performs feature extraction based on the target region, obtaining the feature information corresponding to the target region, and then performs feature matching on the feature information in different original spectral images. The feature matching matrix is ​​used to reflect the feature matching situation between the feature information of different original spectral images. Finally, based on the feature matching matrix, server 104 determines the transformation relationship between at least two original spectral images to be registered, and performs a projection transformation on the original spectral images to be registered based on the transformation relationship to obtain the registered spectral image, thereby improving the registration accuracy of the original spectral images.

[0051] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0052] In one embodiment, such as Figure 2 As shown, a multispectral image registration method is provided, which can be applied to... Figure 1Taking server 104 as an example, it can be understood that this method can also be applied to... Figure 1 The terminal 102 in this embodiment can also be applied to a system including the terminal 102 and the server 104, and is implemented through the interaction between the terminal 102 and the server 104. The method in this embodiment includes:

[0053] Step 201: Obtain at least two original spectral images of the target object to be registered, and locate the target object in each original spectral image to obtain the target region of the target object in each original spectral image.

[0054] In this context, the target object refers to the specific object or feature that needs to be identified, located, or analyzed during image registration. The target object is the common focal point of all original spectral images, i.e., the object captured by all original spectral images. In this embodiment, the target object mainly refers to the bushing of power equipment. As a structure that wraps around and protects the outside of power equipment, the bushing not only provides insulation but also prevents external contaminants (such as dust and moisture) from entering the equipment, thereby protecting its normal operation. Power equipment mainly refers to high-voltage equipment in the power system, and bushing mainly refers to the bushing that wraps around the cables, wires, insulators, and through-wall structures of converter stations of high-voltage equipment.

[0055] In this context, "to be registered" refers to the operation of aligning or registering the original spectral image. Due to factors such as shooting angle, time, and equipment differences, the position, shape, or size of the target object in the original spectral image may be inconsistent. The purpose of registration is to align the target object in the original spectral image to a unified coordinate system or reference base so that subsequent analysis, comparison, or fusion operations can be performed.

[0056] The original spectral image refers to data that combines the spectral and image information of the target object. It can not only display spatial information such as the shape, size, and position of the target object, but also provide images of the spectral characteristics of the target object at different wavelengths. When registering at least two original spectral images, each original spectral image can be a spectral image obtained by using different devices (imaging system, spectrometer, camera, etc.) or from different angles and times for the target object. For example, spectral images obtained by using a binocular camera to capture the target object at the same time from different angles can be used as original spectral images. Or, spectral images of different frames (times) captured by a spectrometer at the same angle can be used as original spectral images.

[0057] In this context, target object localization refers to identifying the location of the target object within the original spectral image. This identification process can be achieved based on specific features of the target object in the original spectral image (such as shape, color, and texture).

[0058] The target region refers to the area in the original spectral image where the target object is located, as determined through the localization process. Identifying the target region ensures that subsequent feature extraction and matching operations are performed on that region, narrowing the scope of data processing. This not only increases the accuracy of data processing but also improves its efficiency.

[0059] For example, the server acquires at least two original spectral images to be registered for the target object (i.e., the device sleeve). For ease of distinction, these two original spectral images are denoted as the first original spectral image and the second original spectral image, respectively. The server locates the target object in the first original spectral image to obtain a first target region of the target object in the first original spectral image, and locates the target object in the second original spectral image to obtain a second target region of the target object in the second original spectral image. It is understood that in some other embodiments, this method is not limited to registering two original spectral images, but can also register two or more original spectral images.

[0060] Step 202: Extract features from each target region to obtain the feature information corresponding to each target region.

[0061] Feature extraction refers to the process of extracting characteristic information that represents the target object from an image range defined by the target region. Feature information refers to the information used to characterize the features and attributes of the target object (i.e., the equipment sleeve). Feature information can be information such as shape, color, texture, and edges, or it can be higher-level features such as the outline of the object and key points, so as to provide a basis for subsequent image registration.

[0062] For example, the server performs feature extraction on the first target region and the second target region respectively to obtain the first feature information corresponding to the first target region and the second feature information corresponding to the second target region.

[0063] Step 203: Perform feature matching on each feature information to obtain the feature matching matrix.

[0064] Feature matching refers to the process of comparing feature information extracted from different original spectral images to determine the correspondence between the positions of the target object in different original spectral images, facilitating subsequent calculation of the transformation relationship between different original spectral images. The feature matching matrix is ​​a matrix used to store the feature matching results.

[0065] For example, the server performs feature matching between the first feature information and the second feature information to obtain a feature matching matrix between the first feature information and the second feature information.

[0066] Step 204: Based on the feature matching matrix, determine the transformation relationship between each original spectral image, and perform projection transformation on each original spectral image based on the transformation relationship to obtain the registered spectral image.

[0067] The transformation relationship refers to the relationship that describes how different original spectral images are converted or aligned with each other; the process of conversion or alignment can be one or more of geometric transformations such as translation, rotation, and scaling.

[0068] In this context, projection transformation refers to the process of mapping an original spectral image from one coordinate system to another. In this embodiment, projection transformation mainly refers to mapping the coordinate system corresponding to the position coordinates of the target object in one original spectral image to the coordinate system corresponding to the position coordinates of the target object in another original spectral image, and vice versa, so as to align or register the two original spectral images in the same coordinate system, thereby obtaining a unified, registered spectral image.

[0069] The registered spectral image refers to the spectral image obtained after position transformation and projection transformation. In the registered spectral image, all target objects in the original spectral images are aligned to the same coordinate system, so that the registered spectral image can be analyzed or processed as a whole.

[0070] For example, the server determines the transformation relationship between the first original spectral image and the second original spectral image based on the feature matching matrix between the first feature information and the second feature information, transforms the first original spectral image or the second original spectral image, and then performs a projection transformation on the first original spectral image and the second original spectral image based on the transformation relationship between them, so as to project the first original spectral image onto the second original spectral image, or project the second original spectral image onto the first original spectral image, thereby obtaining the registered spectral image.

[0071] In other embodiments, when there are two or more original spectral images to be registered, multiple original spectral images can be registered pairwise to obtain a registered intermediate spectral image, and then the intermediate spectral image can be registered pairwise or the intermediate spectral image can be registered with another unprocessed original spectral image to obtain the final registered spectral image.

[0072] In the aforementioned multispectral image registration method, at least two original spectral images of the target object to be registered are acquired, and the target object in each original spectral image is located to obtain the target region of the target object in the corresponding original spectral image. This method can remove background information from the original spectral images, enabling the location of the target object in a non-coaxial spectral image with a complex background. This allows the subsequent registration process to focus on the target region, improving registration accuracy and reducing the data computation load of the subsequent registration process, thus improving registration efficiency. Subsequently, feature extraction is performed on the target objects in each target region to obtain the corresponding feature information of the target objects in each target region. Feature matching is then performed on each feature information to obtain a feature matching matrix. Based on the feature matching matrix, the transformation relationship between each original spectral image is determined, and the original spectral images are projected based on the transformation relationship to obtain the registered spectral image. This method can effectively extract a large number of effective feature points of the target region in the original spectral images to be registered, improving the quality of feature point extraction. Furthermore, based on the correspondence between feature points in different spectral images, the spectral images are registered, which helps to improve the registration accuracy.

[0073] In one embodiment, locating the target object in each original spectral image in step 201 includes:

[0074] For each original spectral image, the target object in the original spectral image is identified by the target localization module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target region of the target object in each original spectral image.

[0075] The image registration model refers to a machine learning model that has been trained and can automatically perform the image registration process. By inputting two different original spectral images, the model processes them and outputs a registered spectral image, thus aligning spectral images acquired at different times, from different viewpoints, or using different devices to the same coordinate system. In this embodiment, the image registration model can be implemented based on one or more of the following algorithms: YOLO series algorithms, R-CNN series algorithms, SuperPoint algorithm (a deep learning-based feature point detection and description algorithm), SuperGlue network (a feature matching network), and SIFT (Scale-invariant feature transform).

[0076] The target localization module refers to the module in the image registration model that locates target objects in the original spectral image. Specifically, as part of the image registration model, the target localization module can employ at least one of the following algorithms: YOLO (You Only Look Once, i.e., object detection algorithm) series (such as YOLOv1~YOLOv5), R-CNN (Region-CNN, i.e., region convolutional neural network), computer vision, etc. By training the target localization module using sample images already labeled with target objects, it can identify and locate different target objects, thereby recognizing the target objects and their positions in the original spectral image.

[0077] In this context, the bounding box refers to the rectangular box that represents the position and size of the target object in the original spectral image. The bounding box coordinates refer to the coordinates of the upper left and lower right corners (or the upper right and lower left corners), or the coordinates of its width, height, and center point, to characterize the position and size of the target object in the original spectral image, facilitating subsequent analysis, processing, and identification of the target object.

[0078] For example, for the first original spectral image and the second original spectral image, the server inputs the first original spectral image and the second original spectral image into the trained image registration model. The target localization module of the image registration model identifies the target objects in the first original spectral image and the second original spectral image respectively. When a target object is identified, the server outputs the first bounding box coordinates of the target object in the first original spectral image and the second bounding box coordinates of the target object in the second original spectral image. The first target region of the target object in the first original spectral image is obtained based on the first bounding box coordinates, and the second target region of the target object in the second original spectral image is obtained based on the second bounding box coordinates.

[0079] In subsequent processing, to ensure image processing accuracy, after obtaining the first and second target regions, the original spectral image can be cropped so that the remaining image contains only the target region, and the spectral image corresponding to the cropped target region is used as the basis for subsequent processing steps.

[0080] In this embodiment, the target localization module of the trained image registration model identifies the target objects in the original spectral images. When a target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target region of the target object in each original spectral image. The target region focuses on the target object to remove complex background interference information in the original spectral image and avoids the extraction of a large amount of interference information during the subsequent feature extraction process, which would affect the accuracy of image registration.

[0081] In one embodiment, the feature information in step 202 includes multiple feature points on the original spectral image that characterize the position of the target object in the original spectral image, and feature descriptors corresponding to each feature point.

[0082] Feature points are points in the original spectral image that represent the characteristics or attributes of a target object. Feature points are typically located at the edges, corners, or textured areas of the target object in the original spectral image; by extracting feature points, the same target object can be found in different original spectral images.

[0083] Among them, feature descriptors are data information used to describe the information of the corresponding spectral image region around the feature point. A feature descriptor is usually a vector that contains statistical information about the brightness, gradient, texture, etc. of the pixels around the feature point. By analyzing the feature descriptors, feature points in different original spectral images can be compared to determine whether two feature points belong to the same target object.

[0084] like Figure 3 As shown, in step 202, feature extraction is performed on each target region to obtain the feature information corresponding to each target region, including:

[0085] Step 301: For each target region, the feature extraction module of the trained image registration model is used to encode the target region to obtain the feature image of the target region.

[0086] The feature extraction module is the module in the image registration model used to extract feature information of the target object from the target region. In specific implementation, the feature extraction module takes the image corresponding to the target region as input, and after processing by the feature extraction module, outputs feature points and feature descriptors corresponding to the target object in the target region.

[0087] Encoding refers to the process of performing convolution, pooling, activation, and concatenation operations on the image data corresponding to the target region based on the feature extraction module, in order to obtain key information in the target region that can reflect the features of the target object.

[0088] In this context, the feature image refers to the image obtained after encoding the target region through a feature extraction module. The feature image contains feature information of the target region. In this embodiment, the feature image consists of several pixels, which contain information such as color and brightness at a certain location. In the feature image, the image is divided into small rectangular regions, each region being a pixel.

[0089] For example, for the first target region and the second target region, the server inputs the spectral images corresponding to the first target region and the second target region into the feature extraction module, and performs feature encoding on the first target region and the second target region through the feature extraction module to obtain the first feature image of the first target region and the second feature image of the second target region.

[0090] Step 302: Decode the feature points of the feature image to obtain the probability that each pixel in the feature image is a candidate point. Filter the candidate points based on the probability to obtain the feature points of the original spectral image.

[0091] Feature point decoding of a feature image refers to the process of converting information in the feature image into candidate points. In this process, the probability of each pixel in the feature image becoming a candidate point is calculated, and the candidate points most likely to become feature points are selected based on these probabilities.

[0092] Candidate points refer to pixels that are selected during the feature point decoding process and are likely to become feature points. These candidate points possess properties that can characterize the target object (such as brightness changes, texture changes, etc.), enabling each candidate point to be stably identified and matched in different images. The probability that a pixel corresponds to a candidate point refers to the likelihood that a certain pixel may become a candidate point.

[0093] For example, the server decodes the feature points of the first feature image and the second feature image respectively to obtain the probability that each pixel in the first feature image is a candidate point and the probability that each pixel in the second feature image is a candidate point. Based on the probability, the candidate points are filtered to obtain the feature points in the corresponding original spectral image.

[0094] Step 303: Decode the feature image to obtain the feature descriptors corresponding to the feature points.

[0095] Descriptor decoding of a feature image refers to the process of converting information in a feature image into feature descriptors corresponding to feature points.

[0096] For example, the server performs descriptor decoding on the first feature image and the second feature image respectively to obtain feature descriptors corresponding to the feature points.

[0097] In an optional embodiment, the feature extraction module, as part of the image registration model, may be based on at least one implementation of methods such as the SuperPoint algorithm, SIFT algorithm, and ORB (Oriented FAST and Rotated BRIEF, i.e., fast feature point detection and descriptor calculation method) to extract effective features belonging to the target object in the target region through automatic learning.

[0098] In this embodiment, the target region is encoded to obtain a feature image of the target region; then the feature image is decoded to obtain the probability that each pixel in the feature image is a candidate point; the candidate points are filtered based on the probability to obtain the feature points of the original spectral image; and the feature image is decoded to obtain the feature descriptors corresponding to the feature points. The whole process is based on the target region, which can achieve efficient extraction of feature points and feature descriptors. At the same time, it can improve the accuracy and stability of feature extraction while reducing computational complexity.

[0099] In one embodiment, such as Figure 4 As shown, in step 203, feature matching is performed on each feature information to obtain a feature matching matrix, including:

[0100] Step 401: For any two target regions, the feature points and feature descriptors are used to determine the registration descriptors corresponding to each feature point in the two target regions through the feature matching module of the image registration model.

[0101] The feature matching module is a component of the image registration model used to compare the similarity of feature points and feature descriptors of target regions in two original spectral images to find corresponding feature point pairs. Specifically, the feature matching module takes feature points and feature descriptors from the two original spectral images of the target region as input, processes them through the feature extraction module, and outputs the matching relationship between the target regions in the two original spectral images.

[0102] The registration descriptor refers to the descriptor obtained by transforming and updating the feature points and their corresponding feature descriptors in each target region, so as to better adapt to the needs of feature matching.

[0103] For example, for any two target regions, the server inputs the feature points and feature descriptors of the first target region and the feature points and feature descriptors of the second target region into the feature matching module of the image registration model. The feature matching module then determines the first registration descriptor corresponding to each feature point in the first target region and the second registration descriptor corresponding to each feature point in the second target region.

[0104] Step 402: Using the feature matching module, based on the registration descriptors corresponding to each feature point in the two target regions, determine the registration score of each registration descriptor in one target region and each registration descriptor in the other target region, and obtain the registration score matrix.

[0105] The registration score is a numerical metric that measures the similarity between two registration descriptors, used to evaluate the degree of matching between the feature points corresponding to the registration descriptors. The registration score matrix is ​​a two-dimensional matrix containing the registration scores of all feature point pairs in the two target regions, used to reflect the potential correspondence between the feature point pairs.

[0106] For example, the server uses a feature matching module to determine the registration score between each first registration descriptor and each second registration descriptor, based on the first and second registration descriptors, to obtain a registration score matrix. Specifically, for the first first registration descriptor, the corresponding registration score is determined with each second registration descriptor and stored in the registration score matrix. Then, for the second first registration descriptor, the corresponding registration score is determined with each second registration descriptor and stored in the registration score matrix, until the registration scores between all first registration descriptors and all second registration descriptors are obtained.

[0107] In determining the registration score, it can be obtained by calculating the inner product of the first registration descriptor and the second registration descriptor.

[0108] Step 403: Using the feature matching module, based on the registration scoring matrix, determine the matching probability between the feature points corresponding to each registration descriptor in one target region and the feature points of each registration descriptor in the other target region, and obtain the feature matching matrix.

[0109] In this context, the matching probability refers to the likelihood that two feature points are considered corresponding or matching points, serving as a metric for selecting the correct feature point pairs in subsequent steps. The feature matching matrix is ​​a two-dimensional matrix containing the matching probabilities of all feature point pairs in two target regions, used to determine which feature point pairs are matched.

[0110] For example, the server uses a feature matching module to determine the matching probability between feature points corresponding to each first registration descriptor in the first target region and feature points of each second registration descriptor in the second target region, based on the registration scoring matrix between the first registration descriptor and the second registration descriptor, and obtains a feature matching matrix.

[0111] In an optional embodiment, the process of obtaining the feature matching matrix is ​​as follows:

[0112] To expand the registration scoring matrix, a row and a column are added to the end of the matrix, resulting in an expanded matrix. The added row and column are used to store feature points that did not match successfully.

[0113] Then, for the extended matrix, two distribution vectors are defined, namely vector a and vector b, where vector a is an N-dimensional vector of all 1s and vector b is an M-dimensional vector of all 1s, indicating that each feature point has the same initial importance. At the same time, a regularization term ε (usually 1) is defined.

[0114] Finally, initialize the feature matching matrix to an all-zero matrix and repeat the following steps until convergence:

[0115] Scaling rows: Divide each element of the expanded matrix by the sum of that row's elements, and then multiply by the corresponding element in vector a.

[0116] Column scaling: Divide each element of the expanded matrix by the sum of that column, and then multiply by the corresponding element of the b vector.

[0117] Check the convergence condition: If the change in the extended matrix is ​​less than a certain threshold, then it is considered to have converged.

[0118] After convergence, the added rows and columns in the expansion matrix are removed to obtain the final feature matching matrix.

[0119] In this embodiment, the corresponding registration descriptor is determined based on the feature points and feature descriptors of the target region, thereby obtaining the registration score between the feature points of the two target regions to construct a feature matching matrix. This allows the feature matching matrix to focus on the target region, ensuring an accurate correspondence between the target objects in the two target regions.

[0120] In one embodiment, such as Figure 5 As shown, in step 401, for any two target regions' feature points and feature descriptors, the feature matching module of the image registration model determines the registration descriptors corresponding to each feature point in the two target regions, including:

[0121] Step 501: For any two target regions, the feature points and feature descriptors of one target region are encoded into a first feature vector by the feature matching module of the image registration model, and the feature points and feature descriptors of the other target region are encoded into a second feature vector.

[0122] Encoding refers to the process of converting feature points and feature descriptors into vector representations.

[0123] The first feature vector is a vector representation obtained by encoding each feature point and feature descriptor in one target region (such as the first target region) through the feature matching module of the image registration model. The first feature vector contains the position and surrounding information of the feature points in the first target region, which is used in the subsequent feature matching process. The second feature vector is similar to the first feature vector; it is a vector representation obtained by encoding each feature point and feature descriptor in another target region (such as the second target region). The second feature vector contains the position and surrounding information of the feature points in the second target region, which is also used in the subsequent feature matching process.

[0124] For example, the server encodes each feature point and feature descriptor in the first target region into a first feature vector and each feature point and feature descriptor in the second target region into a second feature vector using the feature matching module of the image registration model for any two target regions (such as the first target region and the second target region).

[0125] Step 502: Based on the self-attention mechanism, the first feature vectors and the second feature vectors are fused and updated accordingly to obtain the corresponding first intermediate vector and second intermediate vector.

[0126] The self-attention mechanism refers to a mechanism used to capture the relationships between different feature points in an image of the same target region. The self-attention mechanism calculates the similarity between each first feature vector and all other first feature vectors, and calculates the similarity between each second feature vector and all other second feature vectors. Based on their respective similarities, it performs a weighted average on the first and second feature vectors respectively, thereby obtaining the corresponding intermediate vector.

[0127] The first intermediate vector reflects the correlation between the first feature vector and other first feature vectors, that is, the correlation between feature points within the first target region; the second intermediate vector reflects the correlation between the second feature vector and other second feature vectors, that is, the correlation between feature points within the second target region. Both the first and second intermediate vectors retain the feature information of their corresponding feature vectors and incorporate internal correlations, which helps to enhance feature expression.

[0128] Fusion refers to the process of extracting attention from the first feature vector with other first feature vectors and from the second feature vector with other second feature vectors to obtain an embedding representation (or a relevance representation), in order to capture the relevance between features and enhance the expressive power of features.

[0129] Here, "updating" refers to further processing or transforming the fused feature vectors to obtain new first and second intermediate vectors.

[0130] For example, the server performs attention extraction on each first feature vector and other first feature vectors based on a self-attention mechanism and updates the first feature vectors until the end to obtain a first intermediate vector; and performs attention extraction on each second feature vector and other second feature vectors based on a self-attention mechanism and updates the second feature vectors until the end to obtain a second intermediate vector.

[0131] Step 503: Based on the cross-attention mechanism, each first intermediate vector is fused with each second intermediate vector and the first intermediate vector is updated to obtain a first updated vector. Based on the first updated vector, the registration descriptor of the feature points corresponding to each first feature vector is obtained. Based on the cross-sub-attention mechanism, each second intermediate vector is fused with each first intermediate vector and the second intermediate vector is updated to obtain a second updated vector. Based on the second updated vector, the registration descriptor of the feature points corresponding to each second feature vector is obtained.

[0132] The cross-self-attention mechanism refers to a mechanism used to capture the relationship between feature points in the image of the first target region and feature points in the image of the second target region. The cross-self-attention mechanism calculates the similarity between each first intermediate vector and each second intermediate vector, as well as the similarity between each second intermediate vector and each first intermediate vector. Based on their respective similarities, a weighted average is applied to the first and second intermediate vectors to obtain the corresponding update vector.

[0133] The first update vector is used to reflect the correlation between the first intermediate vector and each second intermediate vector, and the second update vector is used to reflect the correlation between the second intermediate vector and each first intermediate vector. That is, the first update vector and the second update vector reflect the correlation between the feature points of the first target region and the feature points of the second target region.

[0134] Fusion refers to the process of extracting attention from the first intermediate vector and each second intermediate vector, as well as from the first intermediate vector and each first intermediate vector, to obtain an embedding representation (or a relevance representation), in order to capture the relevance between features and enhance the expressive power of features.

[0135] Here, "update" refers to further processing or transformation of the fused intermediate vector to obtain a new updated vector.

[0136] For example, the server fuses each first intermediate vector with each second intermediate vector based on a cross-attention mechanism and updates the first intermediate vector to obtain a first updated vector. Based on the first updated vector, it obtains the registration descriptor of the feature points corresponding to each first feature vector. Similarly, based on a cross-sub-attention mechanism, the server fuses each second intermediate vector with each first intermediate vector and updates the second intermediate vector to obtain a second updated vector. Based on the second updated vector, it obtains the registration descriptor of the feature points corresponding to each second feature vector.

[0137] In an optional embodiment, the feature matching module, as part of the image registration model, may employ at least one of the following implementations: RANSAC (Random Sample Consensus), SuperPoint (a self-supervised interest point detection and description algorithm), DeepCompare (a deep learning-based feature matching algorithm), and GMS (Grid-based Motion Statistics, a gray-scale matching-based algorithm for removing mismatched points), to automatically learn and propose mismatched feature point pairs.

[0138] In this embodiment, by employing an attention mechanism to extract attention within the images of the first target region and the second target region, as well as to extract attention between the images of the first target region and the second target region, the information of feature points and feature descriptors of any two target regions can be effectively fused to improve the matching accuracy of feature points.

[0139] In one embodiment, before performing feature matching on each feature information to obtain the feature matching matrix, the method further includes:

[0140] Edge detection is performed on the target region to identify the edge regions of the target region, and the feature information corresponding to the edge regions is removed.

[0141] Edge detection refers to the process of detecting areas of significant brightness change corresponding to a target object within a target region in the original spectral image. Edge detection algorithms detect edges by analyzing changes in the grayscale values ​​of image pixels.

[0142] In this context, the edge region refers to the portion of the target region with significant brightness changes identified by the edge detection algorithm. The edge region corresponds to the outline of the target object, the boundary of the texture, etc. In this embodiment, the edge detection algorithm can be implemented using at least one of the following: the Canny algorithm, the Sobel algorithm, and the Laplacian algorithm.

[0143] Removing feature information from edge regions refers to the process of deleting or ignoring the feature information corresponding to edge regions in an image during image processing in order to reduce noise, simplify the image, and improve processing efficiency.

[0144] For example, the server uses an edge detection algorithm to perform edge detection on the target area, determines the edge areas of the target area, and removes the feature information corresponding to the edge areas.

[0145] Feature matching is performed on each feature information to obtain a feature matching matrix, including:

[0146] Feature matching is performed on the feature information after removing edge regions to obtain the feature matching matrix.

[0147] For example, the server performs feature matching based on the feature information after the region edge to obtain the corresponding feature matching matrix.

[0148] In this embodiment, by using an edge detection algorithm to determine the edge region of the target area and removing the features corresponding to the edge region, errors caused by factors such as irregular edges, noise, or occlusion during image registration can be reduced, thereby improving the accuracy and robustness of registration and obtaining more accurate image registration results.

[0149] In one embodiment, the above multispectral image registration method further includes:

[0150] The registered spectral image is input into the target recognition model to identify the target object, and the recognition result of the target object is output.

[0151] In this context, the target recognition model refers to a model used to identify or classify target objects in an image. The target recognition model can learn the features of target objects by training on a large amount of labeled data. In this embodiment, the target recognition model takes the registered spectral image as input. After processing the registered spectral image, the target recognition model can accurately identify the target objects in the spectral image, facilitating subsequent analysis of defects, performance, etc., of the target objects. Specifically, the target recognition model can employ at least one of the following algorithms: R-CNN series, YOLO (You Only Look Once) series, SSD (SingleShot MultiBox Detector), Mask R-CNN, etc.

[0152] In this embodiment, using the registered spectral image for target object identification and detection helps improve the accuracy of target object identification, thereby providing a reliable basis for subsequent spectral analysis of the target object.

[0153] In a specific application example, when registering the images of the first target region and the second target region, the feature extraction module uses the SuperPoint algorithm, and the feature matching module uses the SuperGlue algorithm. Figure 6 As shown, the feature extraction module has an encoding unit, a feature point decoding unit, and a descriptor decoding unit. The input of the feature extraction module is an image of the target region, and the output is the feature points in the target region and the corresponding feature descriptors. The network architecture of the feature matching module includes an attention map neural network unit and an optimal transport registration unit. The input of the feature matching module is the feature points and feature descriptors of two target region images, and the output is the feature matching relationship between the two target region images.

[0154] The encoding unit includes a convolutional layer, a pooling layer, and a nonlinear activation layer arranged in sequence.

[0155] Convolutional layers are used to extract image features from the target region. A convolutional layer performs a convolution operation on the input target region by sliding a convolutional kernel of a preset size (such as 3×3, 5×5, etc.), thereby obtaining a new feature map. In this embodiment, multiple convolutional layers can be used to extract image features from the target region to obtain deeper features. More specifically, all convolutional layers are expanded, and then convolution operations are performed using convolutional kernels of a preset size, so that the image size of the target region does not change after the convolution operation.

[0156] Pooling layers are used to compress the image corresponding to the input target region, reducing the image size and thus reducing computational cost while extracting the main features of the target object. Pooling layers can be divided into max pooling and average pooling. For example, three 2×2 max pooling layers can be used to change the image size of the target region from H×W to H / 8×W / 8.

[0157] Nonlinear activation layers are used to enable deep neural networks to adapt to complex nonlinear relationships, providing nonlinear mapping capabilities. Activation functions for nonlinear activation layers can be implemented using, but are not limited to, softmax, sigmoid, ReLU (Linear Rectification Function), tanh, etc.

[0158] The feature point decoding unit employs subpixel convolution to convert the feature image output by the encoding unit into feature points in the image. Specifically, the feature point decoding unit processes the input H / 8×W / 8×64 feature image along the channel dimension and reconstructs the feature image, thereby transforming the H / 8×W / 8×64 feature image into an H×W image. The score at each pixel in this image reflects the probability that the pixel is a feature point. Then, feature points are selected through NMS (neuroleptic malignant syndrome) and thresholding.

[0159] The descriptor decoding unit is used to generate feature point descriptors. First, a semi-dense descriptor is generated to reduce the memory usage of the feature extraction module during training and reduce the running time. Then, a complete feature descriptor is obtained through bicubic polynomial interpolation, and the descriptor is normalized through L2 normalization to obtain a feature descriptor that matches the feature points.

[0160] In this process, the attention map neural network unit encodes feature points and feature descriptors into feature vectors. Then, it uses self-attention and cross-attention mechanisms to enhance the feature registration performance of the feature vectors and obtains the registration descriptor. Subsequently, the registration descriptor is calculated by the optimal transmission registration unit to obtain the registration score, and then the registration score matrix is ​​obtained. Finally, the optimal feature score matrix is ​​calculated by the Sinkhorn algorithm.

[0161] In the attention map neural network unit, the input images are the first target region and the second target region. Each image has a feature point P and a corresponding feature descriptor d. First, the feature point P and the feature descriptor are encoded to obtain the feature vector (P, d). The i-th feature point can be represented as P. i = (x, y, c), where c represents the feature point extraction confidence score, and (x, y) represents the coordinates of the feature point; the i-th feature descriptor d i ∈R D Where D is the feature dimension. Then, a graph is used to connect feature points within and between graphs. The graph includes two different types of undirected edges: one type connects feature points within the first target region and the second target region; the other type connects feature points within the first and second target regions. Learnable weights are added to each undirected edge as a mechanism for self-attention and cross-attention extraction. Through iteration of a fixed L-layer graph network, the registration descriptor f for the feature points is finally obtained.

[0162] The optimal transmission registration unit is a module used to calculate the optimal transmission scheme between two feature point sets. First, based on the registration descriptor f obtained from the attention map neural network and the registration scores for each point, a registration score matrix S is obtained. Then, a row and a column are added to the end of the registration score matrix to place unregistered feature points. Next, for a given registration score matrix S, two distribution vectors, a and b, are defined, along with a regularization term (usually 1). Then, the feature matching matrix is ​​initialized, and the rows are repeatedly scaled so that the sum of the rows is a; the columns are scaled so that the sum of the columns is b, until convergence. Finally, the added rows and columns are removed from the converged feature matching matrix to obtain the final optimal feature matching matrix.

[0163] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0164] Based on the same inventive concept, this application also provides a multispectral image registration apparatus for implementing the multispectral image registration method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the multispectral image registration apparatus provided below can be found in the limitations of the multispectral image registration method described above, and will not be repeated here.

[0165] In one embodiment, such as Figure 7 As shown, a multispectral image registration device is provided, including: a target object localization module 701, a feature information extraction module 702, a feature information matching module 703, and an image registration module 704, wherein:

[0166] The target object localization module 701 is used to acquire at least two original spectral images to be registered for the target object, and to locate the target object in each original spectral image to obtain the target region of the target object in each original spectral image.

[0167] The feature information extraction module 702 is used to extract features from each target region to obtain the feature information corresponding to each target region;

[0168] The feature information matching module 703 is used to perform feature matching on each feature information to obtain a feature matching matrix;

[0169] The image registration module 704 is used to determine the transformation relationship between each original spectral image based on the feature matching matrix, and to perform projection transformation on each original spectral image based on the transformation relationship to obtain the registered spectral image.

[0170] In an exemplary embodiment, the target object localization module 701 is further configured to identify the target object in each original spectral image by means of the target localization module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target region of the target object in each original spectral image.

[0171] In an exemplary embodiment, the feature information includes multiple feature points on the original spectral image that characterize the position of the target object in the original spectral image, and feature descriptors corresponding one-to-one with each feature point.

[0172] The feature information extraction module 702 is also used to encode the target region for each target region through the feature extraction module of the trained image registration model to obtain the feature image of the target region; decode the feature image to obtain the probability that each pixel in the feature image is a candidate point; filter the candidate points based on the probability to obtain the feature points of the original spectral image; and decode the feature image to obtain the feature descriptor corresponding to the feature point.

[0173] In an exemplary embodiment, the feature information matching module 703 is further configured to, for any two target regions, determine the registration descriptor corresponding to each feature point in the two target regions through the feature matching module of the image registration model; through the feature matching module, based on the registration descriptor corresponding to each feature point in the two target regions, determine the registration score between each registration descriptor in one target region and each registration descriptor in the other target region, thereby obtaining a registration score matrix; and through the feature matching module, based on the registration score matrix, determine the matching probability between the feature points corresponding to each registration descriptor in one target region and the feature points of each registration descriptor in the other target region, thereby obtaining a feature matching matrix.

[0174] In an exemplary embodiment, the feature information matching module 703 is further configured to, for any two target regions, encode each feature point and feature descriptor in one target region into a first feature vector and encode each feature point and feature descriptor in the other target region into a second feature vector using the feature matching module of the image registration model; fuse each first feature vector and each second feature vector respectively based on a self-attention mechanism and update the first and second feature vectors accordingly to obtain corresponding first and second intermediate vectors; fuse each first intermediate vector with each second intermediate vector respectively based on a cross-attention mechanism and update the first intermediate vector to obtain a first updated vector, and obtain the registration descriptor of the feature point corresponding to each first feature vector based on the first updated vector; and fuse each second intermediate vector with each first intermediate vector respectively based on a cross-sub-attention mechanism and update the second intermediate vector to obtain a second updated vector, and obtain the registration descriptor of the feature point corresponding to each second feature vector based on the second updated vector.

[0175] In an exemplary embodiment, the multispectral image registration device further includes an edge detection module for performing edge detection on the target region, determining the edge regions of the target region, and removing the feature information corresponding to the edge regions.

[0176] The feature information matching module 703 is also used to perform feature matching on the feature information after removing the edge regions to obtain a feature matching matrix.

[0177] In an exemplary embodiment, the multispectral image registration device further includes a target recognition module, which is used to input the registered spectral image into a target recognition model to identify the target object and output the target object recognition result.

[0178] Each module in the aforementioned multispectral image registration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0179] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as raw spectral images, target objects, feature information, feature matching matrices, feature point coordinates, feature descriptors, registration descriptors, and registration scores. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a multispectral image registration method.

[0180] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0181] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multispectral image registration method described above.

[0182] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multispectral image registration method described above.

[0183] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the multispectral image registration method described above.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multispectral image registration method, characterized in that, The method includes: At least two original spectral images of the target object to be registered are acquired, and the target object in each of the original spectral images is located to obtain the target region of the target object in each of the original spectral images; Feature extraction is performed on each of the target regions to obtain the feature information corresponding to each target region; Perform feature matching on each of the aforementioned feature information to obtain a feature matching matrix; Based on the feature matching matrix, the transformation relationship between each of the original spectral images is determined, and the original spectral images are projected and transformed based on the transformation relationship to obtain the registered spectral images; The feature information includes multiple feature points on the original spectral image that characterize the position of the target object in the original spectral image, and feature descriptors corresponding one-to-one with each feature point; The step of extracting features from each of the target regions to obtain feature information corresponding to each target region includes: For each target region, the target region is encoded by the feature extraction module of the trained image registration model to obtain the feature image of the target region; The feature image is decoded to obtain the probability that each pixel in the feature image is a candidate point. The candidate points are then filtered based on the probability to obtain the feature points of the original spectral image. Descriptor decoding is performed on the feature image to obtain the feature descriptor corresponding to the feature point; The step of performing feature matching on each of the aforementioned feature information to obtain a feature matching matrix includes: For any two target regions, the feature points and feature descriptors of the image registration model are used to determine the registration descriptors corresponding to each feature point in the two target regions. Through the feature matching module, based on the registration descriptors corresponding to each feature point in the two target regions, the registration score of each registration descriptor in one of the two target regions and each registration descriptor in the other target region is determined, and a registration score matrix is ​​obtained. The feature matching module determines the matching probability between the feature points corresponding to each registration descriptor in one of the two target regions and the feature points of each registration descriptor in the other target region, based on the registration scoring matrix, thereby obtaining the feature matching matrix. The step of determining the registration descriptor corresponding to each feature point in any two target regions using the feature matching module of the image registration model for the feature points and feature descriptors of any two target regions includes: For any two target regions, the feature points and feature descriptors of the image registration model are encoded into a first feature vector by the feature matching module of one of the target regions, and into a second feature vector by the feature points and feature descriptors of the other target region. Based on the self-attention mechanism, the first feature vectors and the second feature vectors are fused and updated accordingly to obtain the corresponding first intermediate vector and second intermediate vector. Based on the cross-attention mechanism, each first intermediate vector is fused with each second intermediate vector to update the first intermediate vector, resulting in a first update vector. Based on the first update vector, the registration descriptor of the feature points corresponding to each first feature vector is obtained. Based on the cross-sub-attention mechanism, each second intermediate vector is fused with each first intermediate vector to update the second intermediate vector, resulting in a second update vector. Based on the second update vector, the registration descriptor of the feature points corresponding to each second feature vector is obtained.

2. The method according to claim 1, characterized in that, The step of locating the target object in each of the original spectral images to obtain the target region of the target object in each of the original spectral images includes: For each original spectral image, the target object in the original spectral image is identified by the target localization module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target region of the target object in each original spectral image.

3. The method according to any one of claims 1 to 2, characterized in that, Before performing feature matching on each of the aforementioned feature information to obtain the feature matching matrix, the method further includes: Edge detection is performed on the target region to determine the edge regions of the target region, and the feature information corresponding to the edge regions is removed; The step of performing feature matching on each of the aforementioned feature information to obtain a feature matching matrix includes: Feature matching is performed on the feature information after removing the edge regions to obtain a feature matching matrix.

4. The method according to any one of claims 1 to 2, characterized in that, The method further includes: The registered spectral image is input into the target recognition model to identify the target object, and the recognition result of the target object is output.

5. A multispectral image registration device, characterized in that, The device includes: The target object localization module is used to acquire at least two original spectral images to be registered for the target object, and to locate the target object in each of the original spectral images to obtain the target region of the target object in each of the original spectral images; The feature information extraction module is used to extract features from each of the target regions to obtain feature information corresponding to each of the target regions. The feature information matching module is used to perform feature matching on each of the aforementioned feature information to obtain a feature matching matrix; The image registration module is used to determine the transformation relationship between each of the original spectral images based on the feature matching matrix, and to perform projection transformation on each of the original spectral images based on the transformation relationship to obtain the registered spectral images; The feature information includes multiple feature points on the original spectral image that characterize the position of the target object in the original spectral image, and feature descriptors corresponding one-to-one with each feature point; The feature information extraction module is further configured to, for each target region, encode the target region using the feature extraction module of the trained image registration model to obtain a feature image of the target region; decode the feature image to obtain the probability that each pixel in the feature image corresponds to a candidate point; filter the candidate points based on the probability to obtain the feature points of the original spectral image; and decode the feature image to obtain the feature descriptor corresponding to the feature point. The feature information matching module is further configured to, for any two target regions, determine the registration descriptor corresponding to each feature point in the two target regions using the feature matching module of the image registration model, based on the registration descriptors corresponding to each feature point in the two target regions; and, based on the registration descriptors corresponding to each feature point in the two target regions, determine the registration score between each registration descriptor in one target region and each registration descriptor in the other target region, thereby obtaining a registration score matrix; and, based on the registration score matrix, determine the matching probability between the feature points corresponding to each registration descriptor in one target region and the feature points in each registration descriptor in the other target region, thereby obtaining a feature matching matrix. The feature information matching module is further configured to, for any two target regions, encode each feature point and feature descriptor in one of the target regions into a first feature vector and encode each feature point and feature descriptor in the other target region into a second feature vector using the feature matching module of the image registration model; fuse the first feature vectors and the second feature vectors respectively based on a self-attention mechanism and update the first feature vectors and the second feature vectors accordingly to obtain corresponding first intermediate vectors and second intermediate vectors; fuse each first intermediate vector with each second intermediate vector respectively based on a cross-attention mechanism and update the first intermediate vector to obtain a first update vector, and obtain the registration descriptor of the feature point corresponding to each first feature vector based on the first update vector; and fuse each second intermediate vector with each first intermediate vector respectively based on a cross-sub-attention mechanism and update the second intermediate vector to obtain a second update vector, and obtain the registration descriptor of the feature point corresponding to each second feature vector based on the second update vector.

6. The apparatus according to claim 5, characterized in that, The target object localization module is also used to identify the target objects in each original spectral image by using the target localization module of the trained image registration model. When a target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target region of the target object in each original spectral image.

7. The apparatus according to claim 5, characterized in that, It also includes an edge detection module, which is used to perform edge detection on the target area, determine the edge areas of the target area, and remove the feature information corresponding to the edge areas; The feature information matching module is also used to perform feature matching on the feature information after removing the edge regions to obtain a feature matching matrix.

8. The apparatus according to claim 5, characterized in that, It also includes a target recognition module, which is used to input the registered spectral image into the target recognition model to identify the target object and output the target object recognition result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images