Training method, matching method and medium of heterogeneous image matching model

Abstract

The present invention discloses a training method, a matching method and a medium for a heterogeneous image matching model. The training method includes the following steps: A, obtaining a training image data set; B, building a heterogeneous image matching network model; C, randomly selecting and randomly intercepting a first screenshot on a first image and marking key points; processing the first image to obtain a background image; intercepting a second screenshot on a second image and marking key points; D, performing image channel splicing on the background image and the first screenshot to obtain a first mixed image; performing image channel splicing on the background image and the second screenshot to obtain a second mixed image; E, inputting the obtained mixed image into the established network for forward reasoning to obtain multiple key points and corresponding prediction values, and calculating a loss function, and repeating iterations until the loss function converges to a preset value to obtain a trained heterogeneous image matching model, so that the calculation is faster, more accurate and more efficient, and the matching model has better robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a heterogeneous image matching method. Background Art

[0002] Image matching is a key research area in computer vision. It involves finding similar or overlapping regions between two or more images and establishing correspondences between them, thereby providing accurate input for downstream tasks such as image fusion and image stitching. Existing image matching methods, such as SHIFT and ORB, extract image corner points as image feature points. They then use feature point matching algorithms to determine the correspondence between these feature points. They then calculate the homography transformation matrix, which describes the image mapping relationship, to determine the matching relationship between the two images. While these methods are effective for matching images from the same source, they fail when matching images from different sources with vastly different imaging features.

[0003] Heterogeneous images are images generated from different light sources, and their imaging features vary significantly, such as visible light and infrared images. Existing heterogeneous image matching methods require parameter estimation from thousands of feature points, known as the RANSC method, to eliminate outliers. This method also requires manual labeling, making matching time-consuming and inefficient. Summary of the Invention

[0004] In response to the above-mentioned problems existing in the prior art, the present invention provides a training method, a matching method and a medium for a heterogeneous image matching model. The heterogeneous image matching network model is used to extract high-dimensional features of images, directly output the corresponding relationship between key positions of the two images, and achieve end-to-end matching between heterogeneous images.

[0005] In a first aspect, an embodiment of the present application provides a method for training a heterogeneous image matching model, comprising the steps of:

[0006] A, obtain the training image dataset and perform preprocessing;

[0007] B. Build a heterogeneous image matching network model;

[0008] C. Randomly select two different images from the acquired training image dataset, denoted as the first image and the second image, randomly cut out a first rectangular screenshot of random size from the first image, and mark the key points of the first rectangular screenshot; process the first image to obtain a background image; cut out a second rectangular screenshot from the second image at the same position as the first image, and mark the key points of the second rectangular screenshot;

[0009] D. Performing image channel stitching on the background image and the first rectangular screenshot to obtain a first mixed image, wherein channel 1 of the first mixed image is the background image, and channels 2 and 3 are both the first rectangular screenshot; performing image channel stitching on the background image and the second rectangular screenshot to obtain a second mixed image, wherein channel 1 of the second mixed image is the background image, and channels 2 and 3 are both the second rectangular screenshot;

[0010] E. Input the acquired first mixed image and the second mixed image into the established network for forward reasoning to obtain multiple key points and the prediction values corresponding to the multiple key points, and calculate the loss function. Repeat steps C to D iteratively until the loss function converges to a preset value to obtain a trained heterogeneous image matching model.

[0011] In an optional scheme of the first aspect, in step C, the marked key points include the upper left corner point of the rectangular screenshot, the upper right corner point of the rectangular screenshot, the lower left corner point of the rectangular screenshot, the lower right corner point of the rectangular screenshot, the center point of the upper left area of the rectangular screenshot, the center point of the upper right area of the rectangular screenshot, the center point of the lower left area of the rectangular screenshot, the center point of the lower right area of the rectangular screenshot, and the center point of the rectangular screenshot.

[0012] In another optional scheme of the first aspect, in step C, after marking the key points of the rectangular screenshot on the image, the screenshot is also processed to obtain a transformed screenshot, and the image processing method includes random brightness change, random contrast change, random occlusion area and random addition of noise.

[0013] In another optional solution of the first aspect, in step D, before image channel stitching of the background image and the rectangular screenshot is performed, image processing is also performed on the background image; and the rectangular screenshot is scaled to obtain a processed rectangular screenshot having a size consistent with the background image.

[0014] In another optional solution of the first aspect, in step E, the loss function, denoted as L, is calculated as follows:

[0015] L=λ1L con +λ2L loc

[0016]

[0017]

[0018] Among them, L con Indicates the loss value of whether there is a key point at the position, λ1 indicates the weight of the loss value of whether there is a key point at the position, L locrepresents the loss value of the key point coordinates, λ2 represents the weight of the loss value of the key point coordinates, c represents the true value of whether there is a key point, p represents the predicted value of whether there is a key point, x i Indicates the true value of the horizontal coordinate of the i-th key point, y i Represents the true value of the ordinate of the i-th key point, Represents the predicted value of the horizontal coordinate of the i-th key point, Represents the predicted value of the ordinate of the i-th key point.

[0019] In another optional scheme of the first aspect, in step A, the preprocessing includes image cleaning and image conversion, wherein the image cleaning is to remove images without obvious texture features in the data set, and the image conversion is to convert the image into a single-channel grayscale image.

[0020] In another optional scheme of the first aspect, in step B, the constructed heterogeneous image matching network model includes a downsampling area and an upsampling area connected in sequence, and the downsampling area and the upsampling area are connected through a jump connection module; the downsampling area includes multiple levels of downsampling blocks connected in sequence, and each level of downsampling blocks includes a convolution module and a maximum pooling layer connected in sequence, and the upsampling area includes multiple levels of upsampling blocks connected in sequence, and each level of upsampling blocks includes a deconvolution layer and a weighted splicing layer connected in sequence.

[0021] In another optional scheme of the first aspect, in step B, in the constructed heterogeneous image matching network model, the first convolution module in the first-level downsampling block is connected to the weighted splicing layer in the first-level upsampling block, the second convolution module in the second-level downsampling block is connected to the weighted splicing layer in the second-level upsampling block, the third convolution module in the third-level downsampling block is connected to the weighted splicing layer in the third-level upsampling block, the fourth convolution module in the fourth-level downsampling block is connected to the weighted splicing layer in the fourth-level upsampling block, and the maximum pooling layer in the fourth-level downsampling block is connected to the weighted splicing layer in the fifth-level upsampling block.

[0022] In a second aspect, an embodiment of the present application provides a heterogeneous image matching method, which uses a heterogeneous image matching model trained by the above training method to match two images, including the following steps:

[0023] S1, obtaining and preprocessing images to be matched, wherein the images to be matched include a first image and a second image;

[0024] S2, stitching the first image and the second image to obtain a three-channel image, where channel one is the first image, and channels two and three are the second images;

[0025] S3, input the three-channel image into the heterogeneous image matching model for forward reasoning to obtain multiple key points and the corresponding prediction values of the multiple key points;

[0026] S4, if there are more than a preset number of key points whose predicted values are not greater than a preset value, then calculate a homography transformation matrix based on the obtained multiple key points, and transform the first image based on the homography transformation matrix to obtain a transformed image, use the transformed image as the new second image and return to step S2; if there are no such key points, then execute step S5;

[0027] S5, calculating a homography transformation matrix according to the obtained multiple key points, and transforming the three-channel image according to the homography transformation matrix to obtain a matching relationship between the first image and the second image.

[0028] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they can implement the above-mentioned training method of a heterogeneous image matching model.

[0029] The beneficial technical effects of the present invention include:

[0030] The method of the present invention extracts high-dimensional features from images through a deep convolutional neural network, directly outputting the correspondence between key locations between the two images, and achieving end-to-end matching between heterogeneous images. The training data of the deep convolutional neural network can automatically generate labels during the training process, eliminating the need for manual annotation, and the network parameters can achieve self-supervised learning. Compared to existing methods that require parameter estimation to eliminate outliers from thousands of feature points, which is time-consuming and inefficient, the method of the present invention is faster, more accurate, and more efficient, and the heterogeneous image matching model is more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is a flow chart of a training method for a heterogeneous image matching model in the present invention;

[0033] Figure 2 Schematic diagram of the general framework of a heterogeneous image matching model in the present invention;

[0034] Figure 3Schematic diagram of a specific architecture of a heterogeneous image matching model in the present invention;

[0035] Figure 4 This is a flow chart of a heterogeneous image matching method in the present invention;

[0036] Figure 5 Schematic diagram of matching results of a heterogeneous image matching method in the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0038] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application. Different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, even though the embodiment may not be clearly described in the following text.

[0039] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements described without departing from the scope of the present application. Various examples may appropriately omit, replace, or add various processes or components. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, features described in some examples may be combined in other examples.

[0040] Example 1:

[0041] Reference Figure 1 , a training method for a heterogeneous image matching model, comprising the steps of:

[0042] Step A: Obtain and preprocess a training image dataset. This includes open-source image datasets such as ImageNet and COCO, continuous image datasets (i.e., continuous image frames extracted from videos with a 2-second time interval between each frame), and real image datasets (i.e., datasets of similar scenes collected based on actual usage scenarios) to increase the number of difficult samples and improve the method's ability to handle difficult scenarios. Preprocessing includes image cleaning and image conversion. Image cleaning involves removing images without obvious texture features from the dataset, and image conversion involves converting the images into single-channel grayscale images.

[0043] Step B: Build a heterogeneous image matching network model.

[0044] Reference Figure 2 The constructed heterogeneous image matching network model includes a downsampling area and an upsampling area connected in sequence, and the downsampling area and the upsampling area are connected through a jump connection module; the downsampling area includes multiple levels of downsampling blocks connected in sequence, and each level of downsampling block includes a convolution module and a maximum pooling layer connected in sequence, and the upsampling area includes multiple levels of upsampling blocks connected in sequence, and each level of upsampling block includes a deconvolution layer and a weighted splicing layer connected in sequence.

[0045] In the constructed heterogeneous image matching network model, the first convolution module in the first-level downsampling block is connected to the weighted splicing layer in the first-level upsampling block, the second convolution module in the second-level downsampling block is connected to the weighted splicing layer in the second-level upsampling block, the third convolution module in the third-level downsampling block is connected to the weighted splicing layer in the third-level upsampling block, the fourth convolution module in the fourth-level downsampling block is connected to the weighted splicing layer in the fourth-level upsampling block, and the maximum pooling layer in the fourth-level downsampling block is connected to the weighted splicing layer in the fifth-level upsampling block.

[0046] The convolutional modules consist of two sequentially connected convolutional layers and a batch normalization layer (BN layer) along with an activation function. The maximum pooling layer is a convolution operation with a 2*2 kernel and a stride of 2. The skip connection module connects the feature maps of the downsampled area with the feature maps of the upsampled area, enabling direct transfer of feature maps and avoiding information loss during the upsampling process. There are various ways to implement skip connections, such as merging the feature maps of the downsampled area with the feature maps of the upsampled area through concatenation, addition, and multiplication. The role of skip connections is to facilitate information transfer and feature reuse, thereby improving network performance and generalization.

[0047] Reference Figure 3 ,In the constructed network, the training images pass through the downsampling area, the skip connection module, and the upsampling area in sequence.

[0048] Specifically, in the downsampling area, the input training image is a 3-channel image with image width and height of W and H, respectively. In this embodiment, W=640 and H=640. The training image is sequentially convolved with a convolution kernel of C*3*3 and stride 2 in the first convolution layer of the first-level downsampling block to obtain a feature map of C*W / 2*H / 2; convolution is performed with a convolution kernel of (C+1)*3*3 and stride 1 in the second convolution layer to obtain a first feature map of (C+1)*W / 2*H / 2; convolution is performed with a convolution kernel of 3*3 and stride 2 in the maximum pooling layer to obtain a feature map of (C+1)*W / 4*H / 4. The first convolution module includes a first convolution layer and a second convolution layer connected in sequence.

[0049] The training image is convolved in turn through the second convolution module of the second-level downsampling block to obtain the second feature map of (2C+1)*W / 4*H / 4; and convolved through the 3*3 convolution kernel with a stride of 2 in the maximum pooling layer to obtain a feature map of (2C+1)*W / 8*H / 8.

[0050] The training image is convolved in turn through the third convolution module of the third-level downsampling block to obtain the third feature map of (4C+1)*W / 8*H / 8; and convolved through the 3*3 convolution kernel with a stride of 2 in the maximum pooling layer to obtain a feature map of (4C+1)*W / 16*H / 16.

[0051] The training image is convolved in turn through the fourth convolution module of the fourth-level downsampling block to obtain the fourth feature map of (8C+1)*W / 16*H / 16; and convolved through the 3*3 convolution kernel with a stride of 2 in the maximum pooling layer to obtain the fifth feature map of (8C+1)*W / 32*H / 32.

[0052] The training image is convolved with a (16C+1)*3*3 convolution kernel with a step size of 1 in the fifth convolution module of the fifth-level downsampling block to obtain a sixth feature map of (16C+1)*W / 32*H / 32; and convolved with a 3*3 convolution kernel with a step size of 2 in the maximum pooling layer to obtain a feature map of (16C+1)*W / 64*H / 64.

[0053] The bias term is a learnable parameter in a neural network that is used to adjust the output value of the model. Each convolution kernel produces an output feature map, in which each pixel value is obtained by sliding the convolution kernel on the input feature map and performing a weighted summation. The bias term 1 adds a constant value to this basis to adjust the offset of the output feature map.

[0054] In the jump connection module, the training image passes through the convolution layer, deconvolution layer and weighted splicing layer in sequence: convolution is performed with a 32C*3*3 convolution kernel with a step size of 1 in the convolution layer to obtain a feature map of 32C*W / 64*H / 64; deconvolution is performed with a 16C*3*3 convolution kernel with a step size of 2 in the deconvolution layer to obtain a feature map of 16C*W / 32*H / 32; the sixth feature map is obtained by convolution with the fifth convolution module in the five-level downsampling block through the weighted splicing layer. Figure 1 The weighted channels are concatenated together to obtain a concatenated feature map of 16C*W / 32*H / 32.

[0055] Among them, the weighted channel feature splicing is to use the last channel of the sixth feature map (16C+1)*W / 32*H / 32 obtained by convolution of the fifth convolution module in the five-level downsampling block, that is, the bias term as the weight w, and multiply each eigenvalue of the remaining 16C channels by the weight of the corresponding position, and then splice it with the feature map of 16C*W / 32*H / 32 obtained by deconvolution to obtain a spliced feature map of 16C*W / 32*H / 32.

[0056] In the upsampling area, the training image passes through the fifth-level upsampling block and the remaining upsampling blocks in sequence. The fifth-level upsampling block includes a convolution layer and a weighted splicing layer connected in sequence, and the remaining upsampling blocks all include a deconvolution layer and a weighted splicing layer connected in sequence.

[0057] The training image passes through the convolution layer and weighted splicing layer of the fifth-level upsampling block in turn, and is convolved with a convolution kernel of 8C*3*3 and a step size of 2 in the convolution layer to obtain a feature map of 8C*W / 32*H / 32; and is spliced together with the fifth feature map (8C+1)*W / 32*H / 32 to obtain a feature map of 8C*W / 32*H / 32.

[0058] The training image passes through the deconvolution layer and weighted splicing layer of the fourth-level upsampling block in turn, and is deconvolved with the convolution kernel of 8C*3*3 and step size 2 in the deconvolution layer to obtain a feature map of 8C*W / 16*H / 16, which is spliced with the fourth feature map (8C+1)*W / 16*H / 16 to obtain a feature map of 8C*W / 16*H / 16.

[0059] The training image passes through the deconvolution layer and weighted splicing layer of the third-level upsampling block in turn, and is deconvolved with a 4C*3*3 convolution kernel with a step size of 2 in the deconvolution layer to obtain a feature map of 4C*W / 8*H / 8, which is then spliced with the third feature map (4C+1)*W / 8*H / 8 to obtain a feature map of 4C*W / 8*H / 8.

[0060] The training image passes through the deconvolution layer and weighted splicing layer of the second-level upsampling block in turn, and is deconvolved with a convolution kernel of 2C*3*3 and a step size of 2 in the deconvolution layer to obtain a feature map of 2C*W / 4*H / 4, which is then spliced with the second feature map (2C+1)*W / 4*H / 4 to obtain a feature map of 2C*W / 4*H / 4.

[0061] The training image passes through the first deconvolution layer, weighted splicing layer and second deconvolution layer of the first-level upsampling block in turn, and is deconvolved with the convolution kernel of C*3*3 and step size 2 in the first deconvolution layer to obtain a feature map of C*W / 2*H / 2; it is spliced together with the first feature map (C+1)*W / 2*H / 2 to obtain a feature map of C*W / 2*H / 2; it is deconvolved with the convolution kernel of C*3*3 and step size 2 in the second deconvolution layer to obtain the seventh feature map of C*W / *H / .

[0062] In another preferred embodiment of the present invention, the deconvolution operation in the network can also be replaced by a linear interpolation operation.

[0063] The dimension of the seventh feature map is C*W*H. Then, after a convolution operation of 27*3*3 with a step size of 1, an output feature map of dimension 27*W*H, i.e. 27*640*640, is obtained. Each grid of the output feature map is a 27-dimensional vector, divided into 9 groups, and each group of 3 elements is recorded as (p, x, y). p represents the probability of the existence of a key point, and x and y represent the coordinate values of the key point respectively.

[0064] Furthermore, the network can be a mature network such as ResNet, VGG and MatchNet.

[0065] Step C: Randomly select two different images from the acquired training image dataset, denoted as the first image and the second image. A first rectangular screenshot of random size is randomly captured from the first image, and the key points of the first rectangular screenshot are marked. After marking the key points of the rectangular screenshot on the image, the screenshot is further processed to obtain a transformed screenshot. The image processing method includes random brightness changes, random contrast changes, random occlusion areas, and random noise addition. The first image is processed to obtain a background image. A second rectangular screenshot is captured from the second image at the same location as the first image, and the key points of the second rectangular screenshot are marked.

[0066] The marked key points include the upper left corner of the rectangular screenshot (x1, y1), the upper right corner of the rectangular screenshot (x2, y2), the lower left corner of the rectangular screenshot (x3, y3), the lower right corner of the rectangular screenshot (x4, y4), the center point of the upper left area of the rectangular screenshot (x5, y5), the center point of the upper right area of the rectangular screenshot (x6, y6), the center point of the lower left area of the rectangular screenshot (x7, y7), the center point of the lower right area of the rectangular screenshot (x8, y8), and the center point of the rectangular screenshot (x9, y9).

[0067] Step D: performing image channel stitching on the background image and the first rectangular screenshot to obtain a first mixed image. Before performing image channel stitching on the background image and the rectangular screenshot, image processing is performed on the background image and scaling is performed on the rectangular screenshot.

[0068] Perform image processing on the background image: The image processing methods include random brightness changes, random contrast changes, randomly selecting an area for occlusion, randomly adding noise, randomly generating a homography matrix for perspective transformation, translation transformation, rotation transformation, and scaling the image to W*H size to obtain the background image. Perform scaling processing on the rectangular screenshot: The image size after scaling is W*H to obtain a processed rectangular screenshot with the same size as the background image. Channel one of the first mixed image is the background image, and channels two and three are both the first rectangular screenshot. Perform image channel splicing on the background image and the second rectangular screenshot to obtain a second mixed image, and channel one of the second mixed image is the background image, and channels two and three are both the second rectangular screenshot.

[0069] In step E, the first mixed image and the second mixed image are input into the established network for forward reasoning to obtain multiple key points and the prediction values corresponding to the multiple key points, and the loss function is calculated. Steps C to D are iterated repeatedly until the loss function converges to a preset value to obtain a trained heterogeneous image matching model.

[0070] Furthermore, it is determined whether the preset number of training times is reached. If so, the training is stopped; otherwise, steps C to D are iteratively executed.

[0071] The training image datasets include open source image datasets, continuous image datasets, and real image datasets. The training process of the network model includes pre-training and correction training. Pre-training is performed using open source image datasets and continuous image datasets, and correction training is performed using real image datasets. During correction training, all network parameters except the last two layers are frozen so that they are not updated during the iterative training process.

[0072] The loss function, denoted as L, is calculated as:

[0073] L=λ1L con +λ2Lloc

[0074]

[0075]

[0076] Among them, L con Indicates the loss value of whether there is a key point at the position, λ1 indicates the weight of the loss value of whether there is a key point at the position, L loc represents the loss value of the key point coordinates, λ2 represents the weight of the loss value of the key point coordinates, c represents the true value of whether there is a key point, p represents the predicted value of whether there is a key point, x i Indicates the true value of the horizontal coordinate of the i-th key point, y i Represents the true value of the ordinate of the i-th key point, Represents the predicted value of the horizontal coordinate of the i-th key point, Represents the predicted value of the ordinate of the i-th key point.

[0077] Example 2:

[0078] The steps of embodiment 2 are basically the same as those of embodiment 1, except that, in step B, the image width W and the image height H are both integer multiples of 64.

[0079] Example 3: A heterogeneous image matching method

[0080] Reference Figure 5 In order to verify the matching effect of this method, a test data set was obtained to test the method of the present invention. The original image is a RGB three-channel color image with a resolution of 763*1067, recorded as the first image, and referenced Figure 4 Left, the second picture is an infrared image with a resolution of 172*147, recorded as the second image, refer to Figure 4 Right, including the steps:

[0081] Step S1, obtaining and preprocessing the images to be matched, wherein the images to be matched include a first image and a second image;

[0082] Step S2, stitching the first image and the second image to obtain a three-channel image, where channel one is the first image, and channels two and three are the second images;

[0083] Step S3: input the three-channel image into the heterogeneous image matching model for forward reasoning to obtain multiple key points and prediction values corresponding to the multiple key points;

[0084] Step S4: If there are more than a preset number of key points whose predicted values are not greater than a preset value, a homography transformation matrix is calculated based on the obtained multiple key points, and the first image is transformed based on the homography transformation matrix to obtain a transformed image, which is used as the new second image and the process returns to step S2. If there are no such key points, step S5 is executed.

[0085] Step S5, calculate the homography transformation matrix based on the obtained multiple key points, and transform the three-channel image according to the homography transformation matrix to obtain the matching relationship between the first image and the second image. And according to the matching relationship, superimpose the near-infrared test image on the RGB color test image, and the matching result is as follows: Figure 5 shown.

[0086] Figure 5 The size of the circle in the middle represents the confidence level of the matching point. The larger the circle, the higher the confidence level of the matching point. From the matching results in the figure above, we can see that two images of different sizes and different imaging characteristics can be correctly matched, which shows the effectiveness of the method of the present invention.

[0087] Compared with the existing technology, the present invention can cope with the scenario of heterogeneous image matching, and utilizes the powerful image feature extraction ability of deep convolutional neural networks to obtain more accurate results. It can achieve end-to-end implementation from image input to result output, with short time consumption and strong real-time performance.

[0088] Example 4:

[0089] A computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a computer to execute a training method for a heterogeneous image matching model as proposed in Example 1.

[0090] Embodiment 5:

[0091] An electronic device includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform a training method for a heterogeneous image matching model as proposed in Example 1.

[0092] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection of the present invention.

Claims

1. A training method for a heterogeneous image matching model, characterized in that: Including steps: A, obtain the training image dataset and perform preprocessing; B. Build a heterogeneous image matching network model; C. Randomly select two different images from the acquired training image dataset, denoted as the first image and the second image, randomly cut out a first rectangular screenshot of random size from the first image, and mark the key points of the first rectangular screenshot; Processing the first image to obtain a background image; intercepting a second rectangular screenshot at the same position as the first image on the second image, and marking key points of the second rectangular screenshot; D. Performing image channel stitching on the background image and the first rectangular screenshot to obtain a first mixed image, wherein channel 1 of the first mixed image is the background image, and channels 2 and 3 are both the first rectangular screenshot; performing image channel stitching on the background image and the second rectangular screenshot to obtain a second mixed image, wherein channel 1 of the second mixed image is the background image, and channels 2 and 3 are both the second rectangular screenshot; E. Input the acquired first mixed image and the second mixed image into the established network for forward reasoning to obtain multiple key points and predicted values corresponding to the multiple key points, and calculate the loss function. Repeat steps C to D iteratively until the loss function converges to a preset value to obtain a trained heterogeneous image matching model. In step C, the marked key points include the upper left corner of the rectangular screenshot, the upper right corner of the rectangular screenshot, the lower left corner of the rectangular screenshot, the lower right corner of the rectangular screenshot, the center point of the upper left area of the rectangular screenshot, the center point of the upper right area of the rectangular screenshot, the center point of the lower left area of the rectangular screenshot, the center point of the lower right area of the rectangular screenshot, and the center point of the rectangular screenshot; In step C, after marking the key points of the rectangular screenshot on the image, the screenshot is further processed to obtain a transformed screenshot. The image processing method includes random brightness change, random contrast change, random occlusion area and random noise addition.

2. The method for training a heterogeneous image matching model according to claim 1, wherein: In step D, before the background image and the rectangular screenshot are stitched together, the background image is further processed; and the rectangular screenshot is scaled to obtain a processed rectangular screenshot having the same size as the background image.

3. The method for training a heterogeneous image matching model according to claim 1, wherein: In step E, the loss function, denoted as L, is calculated as follows: L=λ1L con +λ2L loc Among them, L con Indicates the loss value of whether there is a key point at the position, λ1 indicates the weight of the loss value of whether there is a key point at the position, L loc represents the loss value of the key point coordinates, λ2 represents the weight of the loss value of the key point coordinates, c represents the true value of whether there is a key point, p represents the predicted value of whether there is a key point, x i Indicates the true value of the horizontal coordinate of the i-th key point, y i Represents the true value of the ordinate of the i-th key point, Represents the predicted value of the horizontal coordinate of the i-th key point, Represents the predicted value of the ordinate of the i-th key point.

4. The method for training a heterogeneous image matching model according to claim 1, wherein: In step A, the preprocessing includes image cleaning and image conversion. The image cleaning is to remove images without obvious texture features in the data set, and the image conversion is to convert the image into a single-channel grayscale image.

5. The method for training a heterogeneous image matching model according to claim 1, wherein: In step B, the constructed heterogeneous image matching network model includes a downsampling area and an upsampling area connected in sequence, and the downsampling area and the upsampling area are connected through a jump connection module; the downsampling area includes multiple levels of downsampling blocks connected in sequence, and each level of downsampling block includes a convolution module and a maximum pooling layer connected in sequence, and the upsampling area includes multiple levels of upsampling blocks connected in sequence, and each level of upsampling block includes a deconvolution layer and a weighted splicing layer connected in sequence.

6. The method for training a heterogeneous image matching model according to claim 5, characterized in that: In step B, in the constructed heterogeneous image matching network model, the first convolution module in the first-level downsampling block is connected to the weighted splicing layer in the first-level upsampling block, the second convolution module in the second-level downsampling block is connected to the weighted splicing layer in the second-level upsampling block, the third convolution module in the third-level downsampling block is connected to the weighted splicing layer in the third-level upsampling block, the fourth convolution module in the fourth-level downsampling block is connected to the weighted splicing layer in the fourth-level upsampling block, and the maximum pooling layer in the fourth-level downsampling block is connected to the weighted splicing layer in the fifth-level upsampling block.

7. A heterogeneous image matching method, characterized in that: Matching two images using a heterogeneous image matching model trained by the training method according to any one of claims 1 to 6, comprising the steps of: S1, obtaining and preprocessing images to be matched, wherein the images to be matched include a first image and a second image; S2, stitching the first image and the second image to obtain a three-channel image, where channel one is the first image, and channels two and three are the second images; S3, input the three-channel image into the heterogeneous image matching model for forward reasoning to obtain multiple key points and the corresponding prediction values of the multiple key points; S4, if there are more than a preset number of key points whose predicted values are not greater than a preset value, then calculate a homography transformation matrix based on the obtained multiple key points, and transform the first image based on the homography transformation matrix to obtain a transformed image, use the transformed image as the new second image and return to step S2; if there are no such key points, then execute step S5; S5, calculating a homography transformation matrix according to the obtained multiple key points, and transforming the three-channel image according to the homography transformation matrix to obtain a matching relationship between the first image and the second image.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the training method of a heterogeneous image matching model as described in any one of claims 1 to 6.

Images