Multispectral image registration method and device, computer equipment and storage medium

Abstract

The invention relates to a multispectral image registration method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring at least two original spectral images to be registered for a target object, and positioning the target object in each original spectral image to obtain a target area of the target object in each original spectral image; performing feature extraction on each target area to obtain feature information corresponding to each target area; performing feature matching on each piece of feature information to obtain a feature matching matrix; and on the basis of the feature matching matrix, determining a transformation relationship among the original spectral images, and performing projection transformation on the original spectral images on the basis of the transformation relationship to obtain a registered spectral image. By adopting the method, the registration precision of the spectral image can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multispectral image registration method, a target recognition method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Insulators or bushings in the power system may become contaminated and aged during long-term use, leading to power grid failures. The use of multispectral imaging technology can achieve fast, accurate, non-contact, and non-destructive monitoring of the status of insulators and insulating bushings, thereby avoiding threats to the stable operation of the power grid.

[0003] Due to the mutual constraints between image spatial resolution and spectral resolution, monocular devices will inevitably reduce their spatial resolution while improving spectral resolution, which does not meet the high-precision requirements of line inspection. In contrast, multi-eye devices can reduce the spectral resolution of a single channel by increasing the number of lenses to ensure the high spatial resolution of the collected image. Since different lenses shoot the same target, even if two lenses are placed side by side, there is still significant optical parallax. At the same time, since the non-coaxial spectral image of the casing usually has a large amount of non-coplanar complex background interference, direct image registration will result in a large number of features outside the casing area being extracted and matched, making it impossible to complete accurate registration of the target area, resulting in low registration accuracy, and then the recognition accuracy of the casing target area is simultaneously reduced. Summary of the invention

[0004] Based on this, it is necessary to provide a multispectral image registration method, device, computer equipment, computer-readable storage medium and computer program product that can improve the registration accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a multispectral image registration method, the method comprising:

[0006] Acquire at least two original spectral images to be registered for the target object, locate the target object in each of the original spectral images, and obtain the target area of ​​the target object in each of the original spectral images;

[0007] Extracting features from each of the target areas to obtain feature information corresponding to each of the target areas;

[0008] Performing feature matching on each of the feature information to obtain a feature matching matrix;

[0009] Based on the feature matching matrix, the transformation relationship between the original spectral images is determined, and based on the transformation relationship, the original spectral images are projected and transformed to obtain the registered spectral images.

[0010] In one embodiment, the positioning of the target object in each of the original spectral images to obtain a target area of ​​the target object in each of the original spectral images includes:

[0011] For each original spectral image, the target object in the original spectral image is identified respectively by the target positioning module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target area of ​​the target object in each of the original spectral images.

[0012] In one of the embodiments, the feature information includes a plurality of feature points on the original spectral image representing the position of the target object in the original spectral image and a feature descriptor corresponding to each feature point;

[0013] The extracting features of each target area to obtain feature information corresponding to each target area includes:

[0014] For each of the target areas, encoding the target area through a feature extraction module of a trained image registration model to obtain a feature image of the target area;

[0015] Decoding the feature points of the feature image to obtain the probability that each pixel point in the feature image corresponds to a candidate point, and screening the candidate points based on the probability to obtain the feature points of the original spectral image;

[0016] Descriptor decoding is performed on the feature image to obtain the feature descriptor corresponding to the feature point.

[0017] In one embodiment, performing feature matching on each of the feature information to obtain a feature matching matrix includes:

[0018] For the feature points and feature descriptors of any two target areas, determining the registration descriptors corresponding to the feature points in the two target areas through the feature matching module of the image registration model;

[0019] By means of the feature matching module, based on the registration descriptors corresponding to the feature points in the two target areas, respectively determining the registration scores of each registration descriptor in one of the two target areas and each registration descriptor in the other target area, to obtain a registration score matrix;

[0020] The feature matching module determines the matching probability between the feature points corresponding to each registration descriptor in one of the two target areas and the feature points of each registration descriptor in the other target area based on the registration score matrix to obtain a feature matching matrix.

[0021] In one embodiment, for the feature points and the feature descriptors of any two target areas, determining the registration descriptors corresponding to the feature points in the two target areas through the feature matching module of the image registration model includes:

[0022] For the feature points and the feature descriptors of any two target areas, encoding each feature point and the feature descriptor in one of the two target areas into a first feature vector, and encoding each feature point and the feature descriptor in the other target area into a second feature vector through the feature matching module of the image registration model;

[0023] Based on the self-attention mechanism, each of the first feature vectors and each of the second feature vectors are respectively fused and the first feature vectors and the second feature vectors are correspondingly updated to obtain a corresponding first intermediate vector and a second intermediate vector;

[0024] Based on the cross-attention mechanism, each of the first intermediate vectors is fused with each of the second intermediate vectors and the first intermediate vectors are updated to obtain a first updated vector, and based on the first updated vector, a registration descriptor of the feature points corresponding to each of the first feature vectors is obtained; and based on the cross-sub-attention mechanism, each of the second intermediate vectors is fused with each of the first intermediate vectors and the second intermediate vectors are updated to obtain a second updated vector, and based on the second updated vector, a registration descriptor of the feature points corresponding to each of the second feature vectors is obtained.

[0025] In one embodiment, before performing feature matching on each of the feature information to obtain a feature matching matrix, the method further includes:

[0026] Performing edge detection on the target area to determine the edge area of ​​the target area, and removing feature information corresponding to the edge area;

[0027] The performing feature matching on each of the feature information to obtain a feature matching matrix includes:

[0028] Feature matching is performed on the feature information after removing the edge area to obtain a feature matching matrix.

[0029] In one embodiment, the method further comprises:

[0030] The registered spectral image is input into the target recognition model to recognize the target object, and the recognition result of the target object is output.

[0031] In a second aspect, the present application further provides a multispectral image registration device, the device comprising:

[0032] A target object positioning module, used to obtain at least two original spectral images to be registered for a target object, and to position the target object in each of the original spectral images to obtain a target area of ​​the target object in each of the original spectral images;

[0033] A feature information extraction module is used to extract features from each target area to obtain feature information corresponding to each target area;

[0034] A feature information matching module is used to perform feature matching on each of the feature information to obtain a feature matching matrix;

[0035] The image registration module is used to determine the transformation relationship between the original spectral images based on the feature matching matrix, and perform a projection transformation on the original spectral images based on the transformation relationship to obtain a registered spectral image.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method when executing the computer program.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0038] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method described above when executed by a processor.

[0039] The above-mentioned multi-spectral image registration method, device, computer equipment, computer-readable storage medium and computer program product obtain at least two original spectral images to be registered for the target object, respectively locate the target object in each original spectral image, and obtain the target area of ​​the target object in the corresponding original spectral image; it can eliminate the background information in the original spectral image, and realize the positioning of the target object of the non-coaxial spectral image of the casing under a complex background, so that the subsequent registration process can be focused on the target area to improve the registration accuracy, and can also reduce the amount of data calculation in the subsequent registration process to improve the registration efficiency; then, by extracting the features of the target objects in each target area, the feature information corresponding to the target object in each target area is obtained; and the feature information is matched to obtain the feature matching matrix; based on the feature matching matrix, the transformation relationship between the original spectral images is determined, and the original spectral images are projected based on the transformation relationship to obtain the registered spectral image, which can effectively extract a large number of effective feature points of the target area in the original spectral image to be registered, can improve the quality of feature point extraction, and based on the correspondence between the feature points in different spectral images, the spectral images are registered, which is conducive to improving the registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A diagram of an application environment of a multispectral image registration method in one embodiment;

[0042] Figure 2 A schematic diagram of a multispectral image registration method according to an embodiment;

[0043] Figure 3 is a schematic diagram of a process of extracting features from each target area in one embodiment;

[0044] Figure 4 A schematic diagram of a process of performing feature matching on each feature information in an embodiment;

[0045] Figure 5 A schematic diagram of a process for determining a registration descriptor in one embodiment;

[0046] Figure 6 A flowchart of registering an image of a first target area with an image of a second target area in an application example;

[0047] Figure 7 is a structural block diagram of a multispectral image registration device in one embodiment;

[0048] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The multispectral image registration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 locates the target object in the original spectral image of the target object, and obtains the position of the target object in the corresponding original spectral image to obtain the target area; then, the server 104 performs feature extraction based on the target area, and after obtaining the feature information corresponding to the target area, the feature information in different original spectral images is feature matched, and the feature matching matrix is ​​used to reflect the feature matching between the feature information of different original spectral images; finally, the server 104 determines the transformation relationship between at least two original spectral images to be registered based on the feature matching matrix, and performs a projection transformation on the original spectral image to be registered based on the transformation relationship to obtain the registered spectral image, so as to improve the registration accuracy of the original spectral image.

[0051] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0052] In one embodiment, Figure 2 As shown, a multispectral image registration method is provided, which is applied to Figure 1The server 104 in FIG. 1 is used as an example to illustrate. It is understandable that the method can also be applied to Figure 1 The terminal 102 in the embodiment may also be applied to a system including the terminal 102 and the server 104, and implemented through interaction between the terminal 102 and the server 104. The method of this embodiment includes:

[0053] Step 201 : acquiring at least two original spectral images to be registered for a target object, locating the target object in each original spectral image, and obtaining a target area of ​​the target object in each original spectral image.

[0054] Among them, the target object refers to a specific object or feature that needs to be identified, located or analyzed in the image registration process. The target object is the common focus of all original spectral images, that is, the object captured by all original spectral images. In this embodiment, the target object mainly refers to the casing of the power equipment. The casing of the power equipment, as a structure that wraps and protects the outside of the power equipment, not only plays an insulating role, but also prevents external dirt (such as dust, moisture, etc.) from entering the inside of the equipment, thereby protecting the normal operation of the equipment. The power equipment mainly refers to the high-voltage equipment in the power system, and the casing mainly refers to the casing wrapped around the cables, wires, insulators, and wall-penetrating structures of the converter station of the high-voltage equipment.

[0055] Among them, the registration refers to the operation that needs to be aligned or registered for the original spectral image. Due to factors such as shooting angle, time, and equipment differences, the position, shape or size of the target object in the original spectral image is inconsistent. The purpose of registration is to align the target object in the original spectral image to a unified coordinate system or reference basis for subsequent analysis, comparison or fusion operations.

[0056] Among them, the original spectral image refers to data that combines the spectral information and image information of the target object, which can not only display the spatial information such as the shape, size and position of the target object, but also provide images of the spectral characteristics of the target object at different wavelengths. When registering at least two original spectral images, each original spectral image can be a spectral image obtained by using different equipment (imaging system, spectrometer, camera, etc.) or photographing the target object from different angles and at different times. For example, the spectral images obtained by photographing the target object from different angles at the same time using a binocular camera are used as the original spectral images, and the spectral images of different frames (times) photographed by a spectrometer at the same angle are used as the original spectral images.

[0057] The positioning of the target object refers to identifying the position of the target object in the original spectral image. During the recognition process, it can be achieved based on the specific features (such as shape, color, texture, etc.) of the target object in the original spectral image.

[0058] The target area refers to the area where the target object is located in the original spectral image, which is determined by the positioning process. By determining the target area, it is possible to ensure that the target area is processed in the subsequent feature extraction and matching operations to narrow the scope of data processing, which can not only increase the accuracy of data processing, but also improve the data processing efficiency.

[0059] Exemplarily, the server obtains at least two original spectral images to be registered for the target object (i.e., the device casing). For the convenience of distinction, the two original spectral images to be registered are taken as an example and are recorded as the first original spectral image and the second original spectral image respectively; the server locates the target object in the first original spectral image to obtain the first target area of ​​the target object in the first original spectral image, and locates the target object in the second original spectral image to obtain the second target area of ​​the target object in the second original spectral image. It is understandable that in some other embodiments, the method is not limited to the registration of two original spectral images, but can also be used to register more than two original spectral images.

[0060] Step 202: extract features from each target area to obtain feature information corresponding to each target area.

[0061] Feature extraction refers to the process of extracting feature information that can represent the target object from the image range defined by the target area. Feature information refers to information used to characterize the characteristics and attributes of the target object (i.e., the device casing). Feature information can be information such as shape, color, texture, edge, etc., or higher-level features such as the object's contour, key points, etc., to provide a basis for subsequent image registration.

[0062] Exemplarily, the server extracts features from the first target area and the second target area respectively to obtain first feature information corresponding to the first target area and second feature information corresponding to the second target area.

[0063] Step 203: perform feature matching on each feature information to obtain a feature matching matrix.

[0064] Among them, feature matching refers to the process of comparing the feature information extracted from different original spectral images to determine the corresponding relationship between the positions of the target object in different original spectral images, so as to facilitate the subsequent calculation of the transformation relationship between different original spectral images. The feature matching matrix refers to the matrix used to store the feature matching results.

[0065] Exemplarily, the server performs feature matching on the first feature information and the second feature information to obtain a feature matching matrix between the first feature information and the second feature information.

[0066] Step 204 : determining the transformation relationship between the original spectral images based on the feature matching matrix, and performing a projection transformation on the original spectral images based on the transformation relationship to obtain a registered spectral image.

[0067] The transformation relationship refers to the relationship that describes how different original spectral images are converted or aligned with each other; the process of mutual conversion or alignment can be one or more geometric transformations such as translation, rotation, and scaling.

[0068] Among them, projection transformation refers to the process of mapping an original spectral image from one coordinate system to another coordinate system. In this embodiment, projection transformation mainly refers to mapping the coordinate system corresponding to the position coordinates of the target object in one original spectral image to the coordinate system corresponding to the position coordinates of the target object in another original spectral image, and vice versa, so as to align or register the two original spectral images in the same coordinate system, thereby obtaining a unified, registered spectral image.

[0069] Among them, the registered spectral image refers to the spectral image obtained after position transformation and projection transformation. In the registered spectral image, all target objects in the original spectral image are aligned to the same coordinate system, so that the registered spectral image can be analyzed or processed as a whole.

[0070] Exemplarily, the server determines the transformation relationship between the first original spectral image and the second original spectral image based on the feature matching matrix between the first feature information and the second feature information, and transforms the first original spectral image or the second original spectral image. Subsequently, the server performs a projection transformation on the first original spectral image and the second original spectral image based on the transformation relationship between the two, so as to project the first original spectral image into the second original spectral image, or project the second original spectral image into the first original spectral image, thereby obtaining the aligned spectral image.

[0071] In some other embodiments, when there are more than two original spectral images to be registered, the multiple original spectral images can be registered in pairs to obtain a registered intermediate spectral image, and then the intermediate spectral images are registered in pairs or the intermediate spectral images are registered with other unprocessed original spectral images to obtain the final registered spectral image.

[0072] In the above multi-spectral image registration method, by acquiring at least two original spectral images to be registered for the target object, the target object in each original spectral image is located respectively, and the target area of ​​the target object in the corresponding original spectral image is obtained; the background information in the original spectral image can be eliminated, and the positioning of the target object of the non-coaxial spectral image of the casing under a complex background can be realized, so that the subsequent registration process can be focused on the target area to improve the registration accuracy, and the data calculation amount of the subsequent registration process can be reduced to improve the registration efficiency; then, by extracting the features of the target objects in each target area, the feature information corresponding to the target object in each target area is obtained; and the feature information is matched to obtain the feature matching matrix; based on the feature matching matrix, the transformation relationship between the original spectral images is determined, and the original spectral image is projected based on the transformation relationship to obtain the registered spectral image, which can effectively extract a large number of effective feature points of the target area in the original spectral image to be registered, can improve the quality of feature point extraction, and based on the correspondence between the feature points in different spectral images, the spectral images are registered, which is conducive to improving the registration accuracy.

[0073] In one embodiment, the positioning of the target object in each original spectral image in step 201 includes:

[0074] For each original spectral image, the target objects in the original spectral image are identified respectively through the target localization module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target area of ​​the target object in each original spectral image.

[0075] Among them, the image registration model refers to a machine learning model that has been trained and can automatically perform the image registration process. By inputting two different original spectral images and processing them through the image registration model, a registered spectral image can be output to achieve the alignment of spectral images acquired at different times, different perspectives or different devices to the same coordinate system. In this embodiment, the image registration model can be implemented based on one or more of the YOLO series of algorithms, the R-CNN series of algorithms, the SuperPoint algorithm (feature point detection and description algorithm based on deep learning), the SuperGlue network (feature matching network), and the SIFT (Scale-invariant feature transform, i.e., scale-invariant feature transformation algorithm).

[0076] Among them, the target positioning module refers to the module in the image registration model that locates the target object in the original spectral image. In specific implementation, as part of the image registration model, the target positioning module can be implemented by using at least one of the YOLO (You Only Look Once, i.e., target detection algorithm) series algorithms (such as YOLOv1~YOLOv5, etc.), R-CNN (Region-CNN‌, i.e., regional convolutional neural network) series algorithms, computer vision and other algorithms. By training the target positioning module with sample images that have been marked with target objects, the target positioning module can recognize and locate different target objects, so as to identify the target objects and their positions in the original spectral image.

[0077] Among them, the bounding box refers to a rectangular box representing the position and size of the target object in the original spectral image, and the bounding box coordinates refer to the coordinates of the upper left corner and lower right corner of the rectangular box (or the coordinates of the upper right corner and lower left corner), or the coordinates of its width, height and center point, so as to characterize the position and size of the target object in the original spectral image, so as to facilitate the subsequent analysis, processing and identification of the target object.

[0078] Exemplarily, for the first original spectral image and the second original spectral image, the server inputs the first original spectral image and the second original spectral image into a trained image registration model, and respectively identifies the target objects in the first original spectral image and the second original spectral image through the target positioning module of the image registration model. When the target object is identified, the first bounding box coordinates of the target object in the first original spectral image and the second bounding box coordinates of the target object in the second original spectral image are output accordingly, and a first target area of ​​the target object in the first original spectral image is obtained based on the first bounding box coordinates, and a second target area of ​​the target object in the second original spectral image is obtained based on the second bounding box coordinates.

[0079] Among them, in order to ensure the processing accuracy of the image during subsequent processing, after obtaining the first target area and the second target area, the original spectral image can be cropped so that the remaining image only contains the target area, and the spectral image corresponding to the cropped target area is used as the basis for subsequent processing steps.

[0080] In this embodiment, the target objects in the original spectral images are identified respectively by the target positioning module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target area of ​​the target object in each original spectral image. The target area is focused on the target object to remove the complex background interference information in the original spectral image, so as to avoid extracting a large amount of interference information in the subsequent feature extraction process and affecting the accuracy of image registration.

[0081] In one embodiment, the feature information in step 202 includes a plurality of feature points on the original spectral image that characterize the position of the target object in the original spectral image and a feature descriptor corresponding to each feature point.

[0082] Among them, feature points refer to points in the original spectral image that can represent the characteristics or attributes of the target object. Feature points are usually located at the edges, corners or texture-rich areas of the target object in the original spectral image; by extracting feature points, the same target object can be found in different original spectral images.

[0083] Among them, the feature descriptor is data information used to describe the information of the spectral image area corresponding to the feature point. The feature descriptor is usually a vector, which contains statistical information such as brightness, gradient, texture, etc. of the pixels around the feature point. By analyzing the feature descriptor, the feature points in different original spectral images can be compared to determine whether two feature points belong to the same target object.

[0084] like Figure 3 As shown, in step 202, feature extraction is performed on each target area to obtain feature information corresponding to each target area, including:

[0085] Step 301 : for each target area, the target area is encoded by a feature extraction module of a trained image registration model to obtain a feature image of the target area.

[0086] The feature extraction module refers to a module in the image registration model that is used to extract feature information of the target object from the target area. In specific implementation, the feature extraction module takes the image corresponding to the target area as input, and after processing by the feature extraction module, outputs feature points and feature descriptors corresponding to the target object in the target area.

[0087] Among them, encoding refers to the process of performing convolution operations, pooling operations, activation, splicing output and other operations on the image data corresponding to the target area in sequence based on the feature extraction module to obtain key information in the target area that can reflect the characteristics of the target object.

[0088] Among them, the feature image refers to the image obtained by encoding the target area through the feature extraction module. The feature image contains the feature information of the target area. In this embodiment, the feature image is composed of a number of pixels, which contain information such as color and brightness of the feature image at a certain position. In the feature image, the image is divided into small rectangular areas, each of which is a pixel.

[0089] Exemplarily, for the first target area and the second target area, the server inputs the spectral images corresponding to the first target area and the second target area into a feature extraction module, and performs feature encoding on the first target area and the second target area through the feature extraction module to obtain a first feature image of the first target area and a second feature image of the second target area.

[0090] Step 302 , decoding the feature points of the feature image, obtaining the probability that each pixel point in the feature image corresponds to a candidate point, and screening the candidate points based on the probability to obtain the feature points of the original spectral image.

[0091] Among them, feature point decoding of the feature image refers to the process of converting the information in the feature image into candidate points. In this process, the probability of each pixel point in the feature image becoming a candidate point is calculated, and then the candidate points most likely to become feature points are screened out based on these probabilities.

[0092] Among them, candidate points refer to pixels that are selected during the feature point decoding process and are likely to become feature points. These candidate points have properties that may be used to characterize the target object (such as brightness changes, texture changes, etc.), so that each candidate point can be stably identified and matched in different images. The probability that a pixel point corresponds to a candidate point refers to the possibility that a certain pixel point may become a candidate point.

[0093] Exemplarily, the server decodes the feature points of the first feature image and the second feature image respectively, obtains the probability that each pixel point in the first feature image corresponds to a candidate point and the probability that each pixel point in the second feature image corresponds to a candidate point, and screens the candidate points based on the probability to obtain the feature points in the corresponding original spectral image.

[0094] Step 303: Decode the feature image and obtain the feature descriptor corresponding to the feature point.

[0095] Decoding the feature image descriptor refers to the process of converting the information in the feature image into a feature descriptor corresponding to the feature point.

[0096] Exemplarily, the server performs descriptor decoding on the first feature image and the second feature image respectively to obtain feature descriptors corresponding to the feature points.

[0097] In an optional embodiment, the feature extraction module, as part of the image registration model, can be implemented based on at least one of the methods such as the SuperPoint algorithm, the SIFT algorithm, and ORB (Oriented FAST and Rotated BRIEF, i.e., fast feature point detection and descriptor calculation method) to extract effective features belonging to the target object in the target area through automatic learning.

[0098] In this embodiment, the target area is encoded to obtain a feature image of the target area; the feature image is then decoded for feature points to obtain the probability that each pixel in the feature image corresponds to a candidate point, the candidate points are screened based on the probability to obtain feature points of the original spectral image, and the feature image is descriptor-decoded to obtain feature descriptors corresponding to the feature points. The entire process is based on the target area and can achieve efficient extraction of feature points and feature descriptors. At the same time, it can also reduce computational complexity while improving the accuracy and stability of feature extraction.

[0099] In one embodiment, Figure 4 As shown, in step 203, feature matching is performed on each feature information to obtain a feature matching matrix, including:

[0100] Step 401 : for feature points and feature descriptors of any two target regions, a feature matching module of an image registration model is used to determine the registration descriptors corresponding to the feature points in the two target regions.

[0101] Among them, the feature matching module refers to the module in the image registration model used to compare the similarity between the feature points and feature descriptors of the target area in the two original spectral images to find the corresponding feature point pairs. In specific implementation, the feature matching module takes the feature points and feature descriptors in the two original spectral images of the target area as input, and after processing by the feature extraction module, outputs the matching relationship between the target areas in the two original spectral images.

[0102] The registration descriptor refers to a descriptor obtained by transforming and updating the feature points and their corresponding feature descriptors in each target area to better meet the needs of feature matching.

[0103] Exemplarily, for the feature points and feature descriptors of any two target areas, the server inputs the feature points and feature descriptors of the first target area and the feature points and feature descriptors of the second target area into the feature matching module of the image registration model, and determines the first registration descriptor corresponding to each feature point in the first target area and the second registration descriptor corresponding to each feature point in the second target area through the feature matching module.

[0104] Step 402, through the feature matching module, based on the registration descriptors corresponding to the feature points in the two target areas, respectively determine the registration scores of each registration descriptor in one of the two target areas with each registration descriptor in the other target area to obtain a registration score matrix.

[0105] The registration score refers to a numerical indicator that measures the similarity between two registration descriptors, and is used to evaluate the matching degree between the feature points corresponding to the registration descriptors. The registration score matrix refers to a two-dimensional matrix containing the registration scores of all feature point pairs in the two target regions, and is used to reflect the potential correspondence between the feature point pairs.

[0106] Exemplarily, the server determines the registration score between each first registration descriptor and each second registration descriptor respectively through the feature matching module and based on the first registration descriptor and the second registration descriptor, and obtains the registration score matrix. Specifically, for the first first registration descriptor, the corresponding registration score is determined with each second registration descriptor and stored in the registration score matrix, and then the corresponding registration score is determined with each second registration descriptor for the second first registration descriptor and stored in the registration score matrix, until the registration scores between all the first registration descriptors and all the second registration descriptors are obtained.

[0107] When determining the registration score, it can be obtained by calculating the inner product of the first registration descriptor and the second registration descriptor.

[0108] Step 403, through the feature matching module, based on the registration score matrix, the matching probability between the feature points corresponding to each registration descriptor in one of the two target areas and the feature points of each registration descriptor in the other target area is determined to obtain a feature matching matrix.

[0109] Among them, the matching probability refers to the possibility that two feature points are considered to be corresponding points or matching points, which is used as a measure to screen out the correct feature point pairs in the subsequent steps. The feature matching matrix refers to a two-dimensional matrix containing the matching probabilities of all feature point pairs in the two target areas, which is used to determine which feature point pairs are matched.

[0110] Exemplarily, the server uses a feature matching module to determine the matching probability between the feature points corresponding to each first registration descriptor in the first target area and the feature points of each second registration descriptor in the second target area based on the registration score matrix between the first registration descriptor and the second registration descriptor, and obtains a feature matching matrix.

[0111] In an optional embodiment, the process of obtaining the feature matching matrix is ​​as follows:

[0112] A row and a column are added to the end of the registration score matrix to expand the registration score matrix, thereby obtaining an expanded matrix; wherein the added row and the added column are used to store feature points that have not been successfully matched.

[0113] Then, for the extended matrix, two distribution vectors are defined, namely a vector and b vector, where a vector is an N-dimensional vector of all 1s and b vector is an M-dimensional vector of all 1s, indicating that each feature point has the same initial importance. At the same time, a regular term ε (usually 1) is defined.

[0114] Finally, initialize the feature matching matrix to an all-zero matrix and repeat the following steps until convergence:

[0115] Scale rows: Divide the elements of each row of the expanded matrix by the sum of the row and multiply by the corresponding element in the a vector.

[0116] Scale columns: Divide the elements of each column of the expanded matrix by the sum of the column and multiply by the corresponding element of the b vector.

[0117] Check the convergence condition: if the change of the expansion matrix is ​​less than a certain threshold, it is considered to have converged.

[0118] After convergence, the added rows and columns in the expansion matrix are removed to obtain the final feature matching matrix.

[0119] In this embodiment, the corresponding registration descriptor is determined based on the feature points and feature descriptors of the target area, and then the registration score between the feature points of the two target areas is obtained to construct a feature matching matrix, so that the feature matching matrix can focus on the target area and ensure the accurate correspondence between the target objects in the two target areas.

[0120] In one embodiment, Figure 5 As shown, in step 401, for the feature points and feature descriptors of any two target regions, the registration descriptors corresponding to the feature points in the two target regions are determined by the feature matching module of the image registration model, including:

[0121] Step 501, for the feature points and feature descriptors of any two target areas, the feature matching module of the image registration model is used to encode the feature points and feature descriptors in one of the two target areas into a first feature vector, and the feature points and feature descriptors in the other target area are encoded into a second feature vector.

[0122] Among them, encoding refers to the process of converting feature points and feature descriptors into vector representations.

[0123] Among them, the first feature vector refers to the vector representation obtained by encoding each feature point and feature descriptor in one of the target areas (such as the first target area) through the feature matching module of the image registration model; the first feature vector contains the location and surrounding information of the feature points in the first target area, which is used in the subsequent feature matching process. The second feature vector is similar to the first feature vector. The second feature vector refers to the vector representation obtained by encoding each feature point and feature descriptor in another target area (such as the second target area); the second feature vector contains the location and surrounding information of the feature points in the second target area, which is used in the subsequent feature matching process.

[0124] Exemplarily, the server encodes the feature points and feature descriptors of any two target areas (such as the first target area and the second target area) into a first feature vector through the feature matching module of the image registration model, and encodes the feature points and feature descriptors in the first target area into a second feature vector.

[0125] Step 502: Based on the self-attention mechanism, each first feature vector and each second feature vector are fused and the first feature vector and the second feature vector are updated accordingly to obtain a corresponding first intermediate vector and a second intermediate vector.

[0126] The self-attention mechanism refers to a mechanism for capturing the relationship between different feature points in an image of the same target area. The self-attention mechanism calculates the similarity between each first feature vector and each other first feature vector, and calculates the similarity between each second feature vector and each other second feature vector, and performs weighted average on the first feature vector and the second feature vector according to their corresponding similarities, thereby obtaining the corresponding intermediate vector.

[0127] Among them, the first intermediate vector is used to reflect the correlation between the first feature vector and other first feature vectors, that is, the correlation between the feature points inside the first target area; the second intermediate vector is used to reflect the correlation between the second feature vector and other second feature vectors, that is, the correlation between the feature points inside the second target area. The first intermediate vector and the second intermediate vector retain the feature information of the corresponding feature vector, and at the same time, they also incorporate the internal correlation, which is conducive to enhancing the expression of features.

[0128] Fusion refers to the process of extracting attention from the first feature vector and other first feature vectors, and from the second feature vector and other second feature vectors to obtain an embedded representation (or correlation representation) to capture the correlation between the features and enhance the expressiveness of the features.

[0129] The updating refers to further processing or transforming the fused feature vector to obtain a new first intermediate vector and a new second intermediate vector.

[0130] Exemplarily, the server extracts attention from each first feature vector and other first feature vectors based on the self-attention mechanism and updates the first feature vector until a first intermediate vector is obtained, and extracts attention from each second feature vector and other second feature vectors based on the self-attention mechanism and updates the second feature vector until a second intermediate vector is obtained.

[0131] Step 503: Based on the cross-attention mechanism, each first intermediate vector is respectively fused with each second intermediate vector and the first intermediate vector is updated to obtain a first updated vector, and based on the first updated vector, a registration descriptor of the feature points corresponding to each first feature vector is obtained; and based on the cross-sub-attention mechanism, each second intermediate vector is respectively fused with each first intermediate vector and the second intermediate vector is updated to obtain a second updated vector, and based on the second updated vector, a registration descriptor of the feature points corresponding to each second feature vector is obtained.

[0132] The cross self-attention mechanism refers to a mechanism for capturing the relationship between feature points in the image of the first target area and feature points in the image of the second target area. The cross self-attention mechanism calculates the similarity between each first intermediate vector and each second intermediate vector and the similarity between each second intermediate vector and each first intermediate vector, and performs weighted average on the first intermediate vector and the second intermediate vector according to their corresponding similarities, thereby obtaining the corresponding update vector.

[0133] The first update vector is used to reflect the correlation between the first intermediate vector and each second intermediate vector, and the second update vector is used to reflect the correlation between the second intermediate vector and each first intermediate vector, that is, the first update vector and the second update vector reflect the correlation between the feature points of the first target area and the feature points of the second target area;

[0134] Fusion refers to the process of extracting attention from the first intermediate vector and the second intermediate vectors, and from the first intermediate vector and the first intermediate vectors to obtain an embedded representation (or correlation representation) to capture the correlation between the features and enhance the expressiveness of the features.

[0135] Here, updating refers to further processing or transforming the fused intermediate vector to obtain a new updated vector.

[0136] Exemplarily, the server fuses each first intermediate vector with each second intermediate vector based on the cross-attention mechanism and updates the first intermediate vector to obtain a first updated vector, and obtains a registration descriptor of the feature points corresponding to each first feature vector based on the first updated vector, and fuses each second intermediate vector with each first intermediate vector based on the cross-sub-attention mechanism and updates the second intermediate vector to obtain a second updated vector, and obtains a registration descriptor of the feature points corresponding to each second feature vector based on the second updated vector.

[0137] In an optional embodiment, the feature matching module, as part of the image registration model, can be implemented using at least one of RANSAC (Random Sample Consensus), SuperPoint (self-supervised interest point detection and description algorithm), DeepCompare (feature matching algorithm based on deep learning), and GMS (Grid-basedMotion Statistics, an algorithm for eliminating mismatched points based on grayscale matching) to propose unmatched feature point pairs through automatic learning.

[0138] In this embodiment, by adopting the attention mechanism to extract attention within the image of the first target area and the image of the second target area, as well as to extract attention between the image of the first target area and the image of the second target area, the information of the feature points and feature descriptors of any two target areas can be effectively fused to improve the matching accuracy of the feature points.

[0139] In one embodiment, before performing feature matching on each feature information to obtain a feature matching matrix, the process further includes:

[0140] Perform edge detection on the target area, determine the edge area of ​​the target area, and remove the feature information corresponding to the edge area.

[0141] Among them, edge detection refers to the process of detecting the places where the brightness of the target object corresponding to the target area in the original spectral image changes significantly. The edge detection algorithm detects the edge by analyzing the gray value changes of the image pixels.

[0142] The edge region refers to a portion of the target region identified by the edge detection algorithm where the brightness changes significantly, and the edge region corresponds to the contour of the target object, the boundary of the texture, etc. In this embodiment, the edge detection algorithm can be implemented by at least one of the Canny algorithm, the Sobel algorithm, and the Laplace algorithm.

[0143] Among them, removing the feature information of the edge area refers to the process of deleting or ignoring the feature information corresponding to the edge area from the image in order to reduce noise, simplify the image, improve processing efficiency, etc. during the image processing process.

[0144] Exemplarily, the server uses an edge detection algorithm to perform edge detection on the target area, determine the edge area of ​​the target area, and remove feature information corresponding to the edge area.

[0145] Perform feature matching on each feature information to obtain a feature matching matrix, including:

[0146] Feature matching is performed on the feature information after removing the edge area to obtain a feature matching matrix.

[0147] Exemplarily, the server performs feature matching based on feature information behind the regional edge region to obtain a corresponding feature matching matrix.

[0148] In this embodiment, by using an edge detection algorithm to determine the edge area of ​​the target area and removing the features corresponding to the edge area, errors caused by factors such as edge irregularities, noise or occlusion during the image registration process can be reduced, thereby improving the accuracy and robustness of the registration and obtaining a more accurate image registration result.

[0149] In one embodiment, the multispectral image registration method further includes:

[0150] The registered spectral image is input into the target recognition model to recognize the target object, and the recognition result of the target object is output.

[0151] Among them, the target recognition model refers to a model used to identify or classify target objects in an image. The target recognition model can learn the characteristics of the target object by training a large amount of labeled data; in this embodiment, the target recognition model takes the registered spectral image as input, and after the target recognition model processes the registered spectral image, it can accurately identify the target object in the spectral image, so as to facilitate the subsequent defect, performance and other analysis of the target object. In specific implementation, the target recognition model can be implemented using at least one of the R-CNN series of algorithms, the YOLO (You Only Look Once) series of algorithms, SSD (SingleShot MultiBox Detector, i.e., single-shot multi-border detection algorithm), Mask R-CNN algorithm, etc.

[0152] In this embodiment, by using the registered spectral image to identify and detect the target object, the accuracy of target object identification is improved, thereby providing a reliable basis for subsequent spectral analysis of the target object.

[0153] In a specific application example, when registering the image of the first target area with the image of the second target area, the feature extraction module is implemented using the SuperPoint algorithm, and the feature matching module is implemented using the SuperGlue algorithm, such as Figure 6 As shown, the feature extraction module has an encoding unit, a feature point decoding unit and a descriptor decoding unit. The input of the feature extraction module is the image of the target area, and the output is the feature points in the target area and the feature descriptors corresponding to the feature points. The network architecture of the feature matching module includes an attention graph neural network unit and an optimal transmission registration unit. The input of the feature matching module is the feature points and feature descriptors of the two target area images, and the output is the feature matching relationship between the two target area images.

[0154] The encoding unit includes a convolutional layer, a pooling layer and a non-linear activation layer arranged in sequence.

[0155] The convolution layer is used to extract the image features of the target area. The convolution layer performs a convolution operation in the input target area by sliding a convolution kernel of a preset size (such as 3×3, 5×5, etc.) to obtain a new feature map. In this embodiment, multiple convolution layers can be used to extract the image features of the target area to obtain deeper features. More specifically, all convolution layers are expanded and then convolution operations are performed using convolution kernels of a preset size, so that the image size of the target area does not change after the convolution operation.

[0156] The pooling layer is used to compress the image corresponding to the input target area to reduce the image size, thereby reducing the amount of calculation and extracting the main features of the target object. The pooling layer can be divided into maximum pooling and average pooling. For example, three 2×2 maximum pooling layers can be used to change the image size of the target area from H×W to H / 8×W / 8.

[0157] The nonlinear activation layer is used to adapt the deep neural network to complex nonlinear relationships and to have nonlinear mapping capabilities. The activation function of the nonlinear activation layer can be implemented using, but is not limited to, softmax, sigmoid, ReLU (Linear Rectification Function, i.e., rectified linear unit), tanh, etc.

[0158] Among them, the feature point decoding unit uses the sub-pixel convolution method to convert the feature image output by the encoding unit into feature points in the image. Specifically, the feature point decoding unit processes the input H / 8×W / 8×64 feature image in the channel dimension and reconstructs the feature image, thereby transforming the H / 8×W / 8×64 feature image into an H×W image. The score at each pixel in the image reflects the probability that the pixel is a feature point, and then the feature points are screened out through NMS (neuroleptic malignant syndrome) and threshold processing.

[0159] Among them, the descriptor decoding unit is used to generate feature point descriptors. First, a semi-dense descriptor is generated to reduce the memory usage of the feature extraction module training and reduce the running time. Then, the complete feature descriptor is obtained through bicubic polynomial interpolation, and the descriptor is normalized through L2 normalization to obtain a feature descriptor that matches the feature point.

[0160] Among them, the attention graph neural network unit encodes the feature points and feature descriptors into feature vectors, and then uses the self-attention mechanism and the cross-attention mechanism to enhance the feature registration performance of the feature vector to obtain the registration descriptor; then, the registration descriptor is calculated by the optimal transmission registration unit to obtain the registration score, and then the registration score matrix is ​​obtained; finally, the optimal feature score matrix is ​​solved by the Sinkhorn algorithm.

[0161] In the attention graph neural network unit, the image of the first target area and the image of the second target area are input. Each image has a feature point P and a corresponding feature descriptor d. First, the feature point P and the feature descriptor are encoded to obtain a feature vector (P, d). The i-th feature point can be represented as P i = (x, y, c), where c represents the extraction confidence of the feature point, (x, y) represents the coordinates of the feature point; the i-th feature descriptor d i ∈R D , where D is the feature dimension. Then the feature points in and between the images are connected in the form of a graph. The graph includes two different undirected edges. One undirected edge connects the feature points in the image of the first target area and the feature points in the image of the second target area. The other undirected edge is the feature points in the image of the first target area and the feature points in the image of the second target area. Learnable weights are added to each undirected edge as a mechanism for self-attention and cross-attention extraction. Through the iteration of a fixed L-layer graph network, the registration descriptor f of the feature points is finally obtained.

[0162] Among them, the optimal transmission registration unit is a module for calculating the optimal transmission scheme between two feature point sets. First, according to the registration descriptor f obtained by the attention graph neural network and the registration of each point, a registration score matrix S is obtained, and then a row and a column are added to the end of the registration score matrix to place the unregistered feature points. Next, for a given registration score matrix S, two distribution vectors, namely a vector and b vector, and a regularization term (usually 1) are defined. Then the feature matching matrix is ​​initialized, and then the rows are repeatedly scaled so that the sum of the rows is a; the columns are scaled so that the sum of the columns is b until convergence, and then the converged feature matching matrix is ​​removed from the added rows and columns to obtain the final optimal feature matching matrix.

[0163] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiment of the present application also provides a multispectral image registration device for implementing the multispectral image registration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multispectral image registration device embodiments provided below can refer to the limitations of the multispectral image registration method above, and will not be repeated here.

[0165] In one embodiment, Figure 7 As shown, a multispectral image registration device is provided, including: a target object positioning module 701, a feature information extraction module 702, a feature information matching module 703 and an image registration module 704, wherein:

[0166] A target object positioning module 701 is used to obtain at least two original spectral images to be registered for a target object, and to position the target object in each original spectral image to obtain a target area of ​​the target object in each original spectral image;

[0167] The feature information extraction module 702 is used to extract features from each target area to obtain feature information corresponding to each target area;

[0168] The feature information matching module 703 is used to perform feature matching on each feature information to obtain a feature matching matrix;

[0169] The image registration module 704 is used to determine the transformation relationship between the original spectral images based on the feature matching matrix, and perform a projection transformation on the original spectral images based on the transformation relationship to obtain a registered spectral image.

[0170] In an exemplary embodiment, the target object localization module 701 is also used to identify the target objects in the original spectral image respectively through the target localization module of the trained image registration model for each original spectral image. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target area of ​​the target object in each original spectral image.

[0171] In an exemplary embodiment, the feature information includes a plurality of feature points on the original spectral image that characterize the position of the target object in the original spectral image and a feature descriptor corresponding to each feature point.

[0172] The feature information extraction module 702 is also used to encode the target area for each target area through the feature extraction module of the trained image registration model to obtain a feature image of the target area; decode the feature points of the feature image to obtain the probability that each pixel point in the feature image corresponds to a candidate point, and screen the candidate points based on the probability to obtain the feature points of the original spectral image; and decode the descriptor of the feature image to obtain the feature descriptor corresponding to the feature point.

[0173] In an exemplary embodiment, the feature information matching module 703 is also used to determine, for the feature points and feature descriptors of any two target areas, the registration descriptors corresponding to the feature points in the two target areas through the feature matching module of the image registration model; through the feature matching module, based on the registration descriptors corresponding to the feature points in the two target areas, respectively determine the registration scores of each registration descriptor in one of the two target areas with each registration descriptor in the other target area to obtain a registration score matrix; through the feature matching module, based on the registration score matrix, determine the matching probability between the feature points corresponding to each registration descriptor in one of the two target areas and the feature points of each registration descriptor in the other target area to obtain a feature matching matrix.

[0174] In an exemplary embodiment, the feature information matching module 703 is also used to encode each feature point and feature descriptor in one of the two target areas into a first feature vector through the feature matching module of the image registration model, and encode each feature point and feature descriptor in the other target area into a second feature vector for the feature points and feature descriptors of any two target areas; based on the self-attention mechanism, each first feature vector and each second feature vector are respectively fused and the first feature vector and the second feature vector are correspondingly updated to obtain the corresponding first intermediate vector and the second intermediate vector; based on the cross-attention mechanism, each first intermediate vector is respectively fused with each second intermediate vector and the first intermediate vector is updated to obtain the first update vector, and based on the first update vector, the registration descriptor of the feature point corresponding to each first feature vector is obtained; based on the cross-sub-attention mechanism, each second intermediate vector is respectively fused with each first intermediate vector and the second intermediate vector is updated to obtain the second update vector, and based on the second update vector, the registration descriptor of the feature point corresponding to each second feature vector is obtained.

[0175] In an exemplary embodiment, the multispectral image registration device further includes an edge detection module for performing edge detection on the target area, determining an edge area of ​​the target area, and removing feature information corresponding to the edge area.

[0176] The feature information matching module 703 is further used to perform feature matching on the feature information after the edge area is removed to obtain a feature matching matrix.

[0177] In an exemplary embodiment, the multispectral image registration device further includes a target recognition module, which is used to input the registered spectral image into a target recognition model to recognize the target object and output the recognition result of the target object.

[0178] Each module in the multispectral image registration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0179] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as original spectral images, target objects, feature information, feature matching matrices, feature point coordinates, feature descriptors, registration descriptors, and registration scores. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multispectral image registration method is implemented.

[0180] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0181] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the multispectral image registration method in the above embodiment are implemented.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multispectral image registration method in the above embodiment are implemented.

[0183] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the multispectral image registration method in the above embodiment when executed by a processor.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multispectral image registration method, characterized in that: The method comprises: Acquire at least two original spectral images to be registered for the target object, locate the target object in each of the original spectral images, and obtain the target area of ​​the target object in each of the original spectral images; Extracting features from each of the target areas to obtain feature information corresponding to each of the target areas; Performing feature matching on each of the feature information to obtain a feature matching matrix; Based on the feature matching matrix, the transformation relationship between the original spectral images is determined, and based on the transformation relationship, the original spectral images are projected and transformed to obtain the registered spectral images.

2. The method according to claim 1, characterized in that The respectively locating the target object in each of the original spectral images to obtain the target area of ​​the target object in each of the original spectral images includes: For each original spectral image, the target object in the original spectral image is identified respectively by the target positioning module of the trained image registration model. When the target object is identified, the bounding box coordinates of the target object in the corresponding original spectral image are output to obtain the target area of ​​the target object in each of the original spectral images.

3. The method according to claim 1, characterized in that The feature information includes a plurality of feature points on the original spectral image representing the position of the target object in the original spectral image and a feature descriptor corresponding to each feature point; The extracting features of each target area to obtain feature information corresponding to each target area includes: For each of the target areas, encoding the target area through a feature extraction module of a trained image registration model to obtain a feature image of the target area; Decoding the feature points of the feature image to obtain the probability that each pixel point in the feature image corresponds to a candidate point, and screening the candidate points based on the probability to obtain the feature points of the original spectral image; Descriptor decoding is performed on the feature image to obtain the feature descriptor corresponding to the feature point.

4. The method according to claim 3, characterized in that The performing feature matching on each of the feature information to obtain a feature matching matrix includes: For the feature points and feature descriptors of any two target areas, determining the registration descriptors corresponding to the feature points in the two target areas through the feature matching module of the image registration model; By means of the feature matching module, based on the registration descriptors corresponding to the feature points in the two target areas, respectively determining the registration scores of each registration descriptor in one of the two target areas and each registration descriptor in the other target area, to obtain a registration score matrix; The feature matching module determines the matching probability between the feature points corresponding to each registration descriptor in one of the two target areas and the feature points of each registration descriptor in the other target area based on the registration score matrix to obtain a feature matching matrix.

5. The method according to claim 4, characterized in that The step of determining the registration descriptors corresponding to the feature points in any two target areas through the feature matching module of the image registration model comprises: For the feature points and the feature descriptors of any two target areas, encoding each feature point and the feature descriptor in one of the two target areas into a first feature vector, and encoding each feature point and the feature descriptor in the other target area into a second feature vector through the feature matching module of the image registration model; Based on the self-attention mechanism, each of the first feature vectors and each of the second feature vectors are respectively fused and the first feature vectors and the second feature vectors are correspondingly updated to obtain a corresponding first intermediate vector and a second intermediate vector; Based on the cross-attention mechanism, each of the first intermediate vectors is fused with each of the second intermediate vectors and the first intermediate vectors are updated to obtain a first updated vector, and based on the first updated vector, a registration descriptor of the feature points corresponding to each of the first feature vectors is obtained; and based on the cross-sub-attention mechanism, each of the second intermediate vectors is fused with each of the first intermediate vectors and the second intermediate vectors are updated to obtain a second updated vector, and based on the second updated vector, a registration descriptor of the feature points corresponding to each of the second feature vectors is obtained.

6. The method according to any one of claims 1 to 5, characterized in that: Before performing feature matching on each of the feature information to obtain a feature matching matrix, the method further includes: Performing edge detection on the target area to determine the edge area of ​​the target area, and removing feature information corresponding to the edge area; The performing feature matching on each of the feature information to obtain a feature matching matrix includes: Feature matching is performed on the feature information after removing the edge area to obtain a feature matching matrix.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The registered spectral image is input into the target recognition model to recognize the target object, and the recognition result of the target object is output.

8. A multispectral image registration device, characterized in that: The device comprises: A target object positioning module, used to obtain at least two original spectral images to be registered for a target object, and to position the target object in each of the original spectral images to obtain a target area of ​​the target object in each of the original spectral images; A feature information extraction module is used to extract features from each target area to obtain feature information corresponding to each target area; A feature information matching module is used to perform feature matching on each of the feature information to obtain a feature matching matrix; The image registration module is used to determine the transformation relationship between the original spectral images based on the feature matching matrix, and perform a projection transformation on the original spectral images based on the transformation relationship to obtain a registered spectral image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images