Infrared image feature extraction method and device, equipment and storage medium

Abstract

The invention relates to the technical field of infrared image processing, and discloses an infrared image feature extraction method, device and equipment and a storage medium, and the method comprises the steps: receiving a target infrared image; inputting the infrared image into a feature coding model, extracting feature points by using a feature point detector, and extracting descriptors by using a feature descriptor encoder; wherein the feature coding model is obtained by performing first joint training on an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point data set and then performing second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image data set; the preset image feature point data set is constructed according to preset shape element images and corresponding feature point true values; the infrared image data set is constructed according to a preset infrared image. By applying the technical scheme of the invention, the recognition precision of the feature points and the feature descriptors in the infrared image can be improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of infrared image processing, and in particular to an infrared image feature extraction method, device, equipment and computer-readable storage medium. Background Art

[0002] The extraction of infrared image features of substation equipment is an important link in substation inspection tasks and infrared image recognition technology. The purpose of infrared image feature extraction is to extract key information from infrared images that can reflect the equipment status for subsequent identification and analysis. Common feature extraction methods include methods based on thermal radiation theory, methods based on physical models, and methods based on computer vision, etc.

[0003] Among them, the method based on computer vision belongs to the field of artificial intelligence, and realizes defect recognition and diagnosis by analyzing and processing power equipment images. This method usually includes steps such as image preprocessing, feature extraction, and classification recognition. In the image preprocessing stage, it is necessary to perform denoising, contrast enhancement, etc. on the collected infrared images to improve the image quality. In the feature extraction stage, various algorithms can be used to extract feature information in the images, such as edge features, texture features, etc. In the classification recognition stage, machine learning or deep learning algorithms are used to classify and recognize the extracted features. The existing feature extraction methods based on computer vision are mainly visual feature extraction methods for visible light images, and the effect of feature extraction achieved by applying them to infrared images is poor, and it is easy to have problems such as unable to recognize feature points. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide an infrared image feature extraction method, device, equipment and computer-readable storage medium for improving the recognition accuracy of feature points and feature descriptors in infrared images.

[0005] According to one aspect of the embodiments of the present invention, an infrared image feature extraction method is provided, and the method includes:

[0006] Receiving a target infrared image;

[0007] Inputting the target infrared image into a feature encoding model, using a feature point detector to extract feature points of the target infrared image, and using a feature descriptor encoder to extract descriptors of the target infrared image;

[0008] Wherein, the feature encoding model is obtained by performing a first joint training on an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point dataset, and then performing a second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset;

[0009] The preset image feature point dataset is constructed according to the preset shape element image and the corresponding feature point ground truth; the infrared image dataset is constructed according to the preset infrared image.

[0010] In an alternative manner, before using the feature point detector to extract the feature points of the target infrared image and using the feature descriptor encoder to extract the descriptors of the target infrared image, it further includes:

[0011] Performing a number of downsampling operations on the target infrared image using the backbone network to obtain a first feature map of the target dimension;

[0012] Wherein, the backbone network includes a number of first connection layers, and the maximum pooling layer is used to connect between every two of the first connection layers. Each of the first connection layers includes a number of sub-connection layers, and each of the sub-connection layers includes a first convolutional layer, a first batch normalization layer, and a first non-linear activation layer connected in sequence;

[0013] Using the first feature map as the input of the feature point detector and the feature descriptor encoder respectively.

[0014] In an alternative manner, the feature point detector includes N second connection layers connected in sequence, a softmax layer, and a reshape operation module; the first (N - 1) of the second connection layers include a second convolutional layer, a second batch normalization layer, and a second non-linear activation layer connected in sequence, and the last second connection layer includes a second convolutional layer and a second batch normalization layer connected in sequence; N is a positive integer and greater than or equal to 2;

[0015] The step of using the feature point detector to extract the feature points of the target infrared image includes:

[0016] Inputting the first feature map into the feature point detector, and passing through a number of the second connection layers and the softmax layer in sequence to obtain a second feature map of the first channel number and the first dimension;

[0017] Wherein, each channel of the second feature map corresponds to an image patch of the target infrared image; the data in each channel of the second feature map represents the probability value that the corresponding image patch is a feature point or the probability that the corresponding image patch has no pixel points;

[0018] Performing a reshape operation on the second feature map of the target channel using the reshape operation module to construct the second feature map of the target channel into a third feature map of the second channel number and the second dimension;

[0019] Wherein, each pixel point in the third feature map represents the probability value that the corresponding pixel point of the target infrared image is a feature point.

[0020] In an alternative manner, the feature descriptor encoder includes M third connection layers connected in sequence, a resize operation module, and an L2 regularization operation module; the first (M - 1) third connection layers include a third convolutional layer, a third batch normalization layer, and a third non-linear activation layer connected in sequence, and the last third connection layer includes a third convolutional layer and a third batch normalization layer connected in sequence; M is a positive integer and M ≥ 2;

[0021] The extracting of the descriptor of the target infrared image by using the feature descriptor encoder includes:

[0022] Inputting the first feature map into the feature descriptor encoder, and passing through several of the third connection layers to obtain a fourth feature map with a third number of channels and a third dimension;

[0023] Performing a resize operation on the fourth feature map by using the resize operation module to obtain a fifth feature map with a fourth number of channels and a fourth dimension;

[0024] Performing an L2 regularization operation on the fifth feature map by using the L2 regularization operation module to obtain a sixth feature map with a fifth number of channels and a fifth dimension;

[0025] Wherein, the vector in each channel of the sixth feature map represents the descriptor of the pixel point corresponding to the target infrared image.

[0026] In an alternative manner, before inputting the target infrared image into the feature encoding model, using the feature point detector to extract the feature points of the target infrared image, and using the feature descriptor encoder to extract the descriptor of the target infrared image, it further includes:

[0027] Constructing an initial feature point detector and an initial feature descriptor encoder;

[0028] Performing a first joint training on the initial feature point detector and the initial feature descriptor encoder based on a preset image feature point dataset;

[0029] Performing a second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset to obtain a trained feature point detector and a trained feature descriptor encoder;

[0030] Combining the backbone network with the trained feature point detector and the trained feature descriptor encoder to obtain a feature encoding model.

[0031] In an alternative manner, performing a first joint training on the initial feature point detector and the initial feature descriptor encoder based on a preset image feature point dataset includes:

[0032] Obtain a first original image from a preset image feature point dataset;

[0033] Perform transformation processing on the first original image to obtain a first transformed image;

[0034] The transformation processing includes: randomly one or more of cropping, translation, scaling, rotation, and perspective;

[0035] Input the first original image and the first transformed image into an initial feature descriptor encoder respectively, and obtain a first feature descriptor set of the first original image and a second feature descriptor set of the first transformed image respectively;

[0036] Calculate a first loss function according to the distances of the corresponding feature descriptors in the first feature descriptor set and the second feature descriptor set;

[0037] Input the first original image into an initial feature point detector to obtain a sixth feature map of several channels; each channel of the sixth feature map corresponds to an image patch of the first original image; the data in each channel of the sixth feature map represents the probability value that the corresponding image patch in the first original image is a feature point or the probability that the corresponding image patch has no pixel points;

[0038] Input the first transformed image into an initial feature point detector to obtain a seventh feature map of several channels; each channel of the seventh feature map corresponds to an image patch of the first transformed image; the data in each channel of the seventh feature map represents the probability value that the corresponding image patch in the first transformed image is a feature point or the probability that the corresponding image patch has no pixel points;

[0039] Calculate a second loss function according to the feature point ground truth of the first original image and the sixth feature map;

[0040] Calculate a third loss function according to the feature point ground truth of the first transformed image and the seventh feature map;

[0041] Obtain a fourth loss function according to the first loss function, the second loss function, and the third loss function;

[0042] Optimize the preset parameters of the initial feature point detector and the initial feature descriptor encoder by using the fourth loss function to obtain a first jointly trained feature point detector and feature descriptor encoder.

[0043] In an alternative manner, performing second joint training on the first jointly trained feature point detector and feature descriptor encoder based on an infrared image dataset includes:

[0044] Obtain the second original image in the infrared image dataset;

[0045] Perform transformation processing on the second original image to obtain a second transformed image;

[0046] The transformation processing includes: randomly one or more of cropping, translation, scaling, rotation, and perspective;

[0047] Input the second original image and the second transformed image into the first jointly trained feature descriptor encoder respectively, and obtain a third feature descriptor set of the second original image and a fourth feature descriptor set of the second transformed image respectively;

[0048] Calculate a fifth loss function according to the distances of the corresponding feature descriptors in the third feature descriptor set and the fourth feature descriptor set;

[0049] Input the second original image into the first jointly trained feature point detector to obtain an eighth feature map of several channels, and determine the true value of the feature points of the second original image; each channel of the eighth feature map corresponds to an image block of the second original image; the data in each channel of the eighth feature map represents the probability value that the corresponding image block is a feature point or the probability that the corresponding image block has no pixel points;

[0050] Input the second transformed image into the initial feature point detector to obtain a ninth feature map of several channels, and determine the true value of the feature points of the second transformed image; each channel of the ninth feature map corresponds to an image block of the second transformed image; each channel of the ninth feature map corresponds to an image block of the second transformed image; the data in each channel of the ninth feature map represents the probability value that the corresponding image block is a feature point or the probability that the corresponding image block has no pixel points;

[0051] Calculate a sixth loss function according to the true value of the feature points of the second original image and the eighth feature map;

[0052] Calculate a seventh loss function according to the true value of the feature points of the second transformed image and the ninth feature map;

[0053] Obtain an eighth loss function according to the sixth loss function and the seventh loss function;

[0054] Use the eighth loss function to optimize the preset parameters of the first jointly trained feature point detector and the first jointly trained feature descriptor encoder, and obtain a second jointly trained feature point detector and feature descriptor encoder.

[0055] An infrared image feature extraction method proposed in this application uses the basic graphic feature point data to perform the first joint training on the initial feature point detector and the feature descriptor encoder, and uses the feature point detector and the feature descriptor encoder after the first joint training to self-supervisedly generate the feature point ground truth and the feature descriptor of the preset infrared image, so as to construct a preset infrared image feature point data set. Based on the preset infrared image feature point data set, the second joint training is performed on the feature point detector and the feature descriptor encoder after the first joint training to obtain the trained feature point detector and the feature descriptor encoder. The feature encoding model constructed according to the trained feature point detector and the feature descriptor encoder in this application can more accurately detect the feature points and the feature descriptors in the infrared image.

[0056] According to another aspect of the embodiments of the present invention, an infrared image feature extraction device is provided, including: a receiving module and a feature extraction module;

[0057] The receiving module is used to receive the target infrared image;

[0058] The feature extraction module is used to input the target infrared image into the feature encoding model, extract the feature points of the target infrared image by using the feature point detector, and extract the descriptor of the target infrared image by using the feature descriptor encoder;

[0059] Wherein, the feature encoding model is obtained by performing the first joint training on the initial feature point detector and the initial feature descriptor encoder based on the preset image feature point data set, and then performing the second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on the infrared image data set;

[0060] The preset image feature point data set is constructed according to the preset shape element image and the corresponding feature point ground truth; the infrared image data set is constructed according to the preset infrared image.

[0061] According to another aspect of the embodiments of the present invention, an infrared image feature extraction device is provided, including: a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface complete the communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operations of the infrared image feature extraction method as described in any one of the above.

[0062] According to still another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, and at least one executable instruction is stored in the storage medium, and the executable instruction enables the infrared image feature extraction device / device to perform the operations of the infrared image feature extraction method as described in any one of the above.

[0063] The above description is only an overview of the technical solution of the embodiments of the present invention. In order to be able to understand the technical means of the embodiments of the present invention more clearly, it can be implemented according to the content of the description. And in order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the following specifically illustrates the specific implementation manners of the present invention. Description of the Drawings

[0064] The drawings are only used to illustrate the embodiments and are not considered as a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0065] Figure 1 A flowchart of an embodiment of the infrared image feature extraction method provided by the present invention is shown;

[0066] Figure 2 A network structure diagram of the feature encoding model of an embodiment of the infrared image feature extraction method provided by the present invention is shown;

[0067] Figure 3 An example diagram of some basic element images of an embodiment of the infrared image feature extraction method provided by the present invention is shown;

[0068] Figure 4 A structural diagram of an embodiment of the infrared image feature extraction device provided by the present invention is shown;

[0069] Figure 5 A structural diagram of an embodiment of the infrared image feature extraction device provided by the present invention is shown. Detailed Description of the Embodiments

[0070] The exemplary embodiments of the present invention will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein.

[0071] Figure 1 A flowchart of an embodiment of the infrared image feature extraction method of the present invention is shown. This method is executed by an infrared image feature extraction device and can be used for feature extraction of substation equipment infrared images.

[0072] As Figure 1 shown, the method includes the following steps:

[0073] Step 110: Receive a target infrared image.

[0074] Step 120: Input the target infrared image into a feature encoding model, extract the feature points of the target infrared image by using a feature point detector, and extract the descriptors of the target infrared image by using a feature descriptor encoder;

[0075] Among them, the feature encoding model is obtained by first jointly training an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point dataset, and then secondarily jointly training the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset;

[0076] The preset image feature point dataset is constructed according to a preset shape element image and corresponding feature point ground truth; the infrared image dataset is constructed according to a preset infrared image.

[0077] In some embodiments of the present invention, before using the feature point detector to extract the feature points of the target infrared image and using the feature descriptor encoder to extract the descriptors of the target infrared image, it further includes:

[0078] Performing a plurality of downsampling operations on the target infrared image using a backbone network to obtain a first feature map of a target dimension;

[0079] Among them, the backbone network includes a plurality of first connection layers, and a max pooling layer is used to connect between every two of the first connection layers. Each of the first connection layers includes a plurality of sub-connection layers, and each of the sub-connection layers includes a first convolutional layer, a first batch normalization layer, and a non-linear activation layer connected in sequence;

[0080] Using the first feature map as the input of the feature point detector and the feature descriptor encoder respectively.

[0081] Preferably, the backbone network (backbone network) is a VGG-style encoder for reducing the image dimension. The backbone network includes 4 first connection layers, and a max pooling layer (max_pooling) is used to connect between every two of the first connection layers. Each of the first connection layers includes 2 sub-connection layers, and each of the sub-connection layers includes a 3*3 first convolutional layer (conv2d), a batch normalization layer (BN), and a ReLU non-linear activation layer (ReLU) connected in sequence. Among them, the stride of each max pooling layer is 2, and non-overlapping pooling operations are performed. At the same time, each max pooling layer reduces the feature map to 1 / 2 of the original.

[0082] In some embodiments of the present invention, before inputting the target infrared image into the feature encoding model, using the feature point detector to extract the feature points of the target infrared image, and using the feature descriptor encoder to extract the descriptors of the target infrared image, it further includes:

[0083] Constructing an initial feature point detector and an initial feature descriptor encoder;

[0084] Perform the first joint training on the initial feature point detector and the initial feature descriptor encoder based on a preset image feature point dataset;

[0085] Perform the second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset;

[0086] Combine the backbone network with the feature point detector and the feature descriptor encoder after the second joint training to obtain a feature encoding model.

[0087] Preferably, the initial feature point detector includes two second connection layers connected in sequence, a softmax layer (softmax), and a reshape operation module (reshape); the first of the second connection layers includes a 3*3 second convolutional layer (conv2d), a second batch normalization layer (BN), and a second non-linear activation layer (ReLU) connected in sequence, and the second connection layer includes a 3*3 second convolutional layer (conv2d) and a second batch normalization layer (BN) connected in sequence, without a non-linear activation layer.

[0088] The initial feature descriptor encoder includes: two third connection layers connected in sequence, a resize operation module (resize), and an L2 regularization operation module (L2-norm); the first of the third connection layers includes a 3*3 third convolutional layer (conv2d), a third batch normalization layer (BN), and a third non-linear activation layer (ReLU) connected in sequence, and the last third connection layer includes a 3*3 third convolutional layer (conv2d) and a third batch normalization layer (BN) connected in sequence, without a non-linear activation layer.

[0089] Perform the first joint training on the initial feature point detector and the initial feature descriptor encoder based on a preset image feature point dataset; perform the second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset to obtain a trained feature point detector (abbreviation, feature point detector) and a trained feature descriptor decoder (abbreviation, feature descriptor decoder);

[0090] Specifically, the device connects the backbone network to the feature point detector and the feature descriptor encoder that have completed training respectively, serving as the feature encoding model. Among them, the initial feature point detector and the initial feature descriptor encoder are subjected to the first joint training to optimize and adjust the parameters in the initial feature point detector and the initial feature descriptor encoder; the feature point detector and the feature descriptor encoder after the first joint training are subjected to the second joint training to further optimize and adjust the parameters in the feature point detector and the feature descriptor encoder after the first joint training, and the structure of the feature point detector after completion of training remains unchanged, that is, the structure of the feature point detector after completion of training is the same as that of the initial feature point detector, and the structure of the feature descriptor encoder after completion of training is the same as that of the initial feature point detector.

[0091] In some embodiments of the present invention, the feature point detector includes N second connection layers connected in sequence, a softmax layer, and a reshape operation module; the first (N - 1) of the second connection layers include a second convolutional layer, a second batch normalization layer, and a second non-linear activation layer connected in sequence, and the last second connection layer includes a second convolutional layer and a second batch normalization layer connected in sequence; N is a positive integer and N≥2;

[0092] The extraction of the feature points of the target infrared image by using the feature point detector includes:

[0093] Input the first feature map into the feature point detector, and successively pass through several of the second connection layers and the softmax layer to obtain a second feature map with the first number of channels and the first dimension;

[0094] Among them, each channel of the second feature map corresponds to an image block of the target infrared image; the data in each channel of the second feature map represents the probability value that the corresponding image block is a feature point or the probability that the corresponding image block has no pixel points;

[0095] Use the reshape operation module to perform a reshape operation on the second feature map of the target channel, and construct the second feature map of the target channel into a third feature map with the second number of channels and the second dimension;

[0096] Among them, each pixel point in the third feature map represents the probability value that the corresponding pixel point of the target infrared image is a feature point.

[0097] In some embodiments of the present invention, the feature descriptor encoder includes M third connection layers connected in sequence, a resize operation module, and an L2 regularization operation module; the first (M-1) of the third connection layers include a third convolutional layer, a third batch normalization layer, and a third non-linear activation layer connected in sequence, and the last third connection layer includes a third convolutional layer and a third batch normalization layer connected in sequence; M is a positive integer and M≥2;

[0098] The extraction of the descriptor of the target infrared image by using the feature descriptor encoder includes:

[0099] Input the first feature map into the feature descriptor encoder, and after passing through several of the third connection layers, obtain a fourth feature map with a third number of channels and a third dimension;

[0100] Use the resize operation module to perform a resize operation on the fourth feature map to obtain a fifth feature map with a fourth number of channels and a fourth dimension;

[0101] Use the L2 regularization operation module to perform an L2 regularization operation on the fifth feature map to obtain a sixth feature map with a fifth number of channels and a fifth dimension;

[0102] Wherein, the vector in each channel of the sixth feature map represents the descriptor of the pixel point corresponding to the target infrared image.

[0103] Exemplarily, the network structure of the feature encoding model is as Figure 2 shown. The feature encoding model includes: an input layer and a backbone network. Two branches are connected behind the backbone network. One branch is a trained feature point detector (abbreviation: feature point detector), and the other branch is a trained feature descriptor decoder (abbreviation: feature descriptor decoder).

[0104] Wherein, the backbone network includes 4 first connection layers, and a max_pooling layer is used to connect every two first connection layers. Each of the first connection layers includes 2 sub-connection layers, and each of the sub-connection layers includes a 3*3 first convolutional layer (conv2d), a batch normalization layer (BN), and a non-linear activation layer (ReLU) connected in sequence. Wherein, the stride of each max_pooling layer is 2, the ReLU non-linear activation layer in the first two sub-connection layers is 64ReLU, and the ReLU non-linear activation layer in the first two sub-connection layers is 128ReLU. Among them, 64ReLU represents a ReLU activation layer with an input channel number of 64; 128ReLU represents a ReLU with an input channel number of 128.

[0105] The feature point detector includes: two second connection layers connected in sequence, a softmax layer (softmax), and a reshape operation module (reshape); the first of the second connection layers includes a 3*3 second convolutional layer (conv2d), a second batch normalization layer (BN), and a second non-linear activation layer (ReLU) connected in sequence, and the second connection layer includes a 3*3 second convolutional layer (conv2d) and a second batch normalization layer (BN) connected in sequence, without a non-linear activation layer. Among them, the number of channels of the first connection layer is 256, and the number of channels of the second connection layer is 65.

[0106] The feature descriptor encoder includes: two third connection layers connected in sequence, a resize operation module (resize), and an L2 regularization operation module (L2-norm); the first of the third connection layers includes a 3*3 third convolutional layer (conv2d), a third batch normalization layer (BN), and a third non-linear activation layer (ReLU) connected in sequence, and the last third connection layer includes a 3*3 third convolutional layer (conv2d) and a third batch normalization layer (BN) connected in sequence, without a non-linear activation layer. Among them, the number of channels of the first connection layer is 256, and the number of channels of the second connection layer is 256.

[0107] Specifically, the device inputs a target image (image) with a height of H and a width of W into the feature encoding model. After the backbone network performs 3 downsampling operations on the target image, a first feature map with a dimension of 128*H / 8*W / 8 is obtained, denoted as H c =H / 8, W c =W / 8. Among them, H c is the height of the first feature map and W c is the width of the first feature map The device uses the first feature map as the input of the feature point detector and the feature descriptor encoder respectively.

[0108] In the branch of the feature point detector, the device uses the feature point detector to input the first feature map into two second connection layers and a softmax layer in sequence, and obtains a second feature map with a first channel number of 65, a height of H / 8, and a width of W / 8, denoted as X. Among them, the feature point detector divides the first feature map into 8*8 image patches (patch). Each of the first 64 channels in the 65 channels of the feature point detector corresponds to a different image patch, and the data in these 64 channels represents the probability value that the corresponding image patch is a feature point, and the remaining 1 channel represents the probability that the current image patch has no feature points.

[0109] Use the reshape operation module to perform a reshape operation on the data of the first 64 channels of the second feature map X, and construct the data of the first 64 channels into a third feature map with a dimension of 1*H*W, denoted as X H*W ; wherein, each pixel point in the third feature map represents the probability value that the image block corresponding to the target image is a feature point, and thus the feature points of the target infrared image can be determined.

[0110] In the branch of the feature descriptor encoder, the device uses the feature descriptor encoder to input the first feature map into two third connection layers in sequence to obtain a fourth feature map with a dimension of 256*H / 8*W / 8, denoted as D. After the fourth feature map D undergoes a resize operation through the resize operation module, the fourth feature map D is constructed into a fifth feature map with a dimension of 256*H*W; after the fifth feature map undergoes an L2 regularization operation through the L2-norm operation module, a sixth feature map with a dimension of 256*H*W is output, denoted as D H*W , and the vectors in the 256 channels of the sixth feature map are used as the descriptors of the corresponding pixel points in the target infrared image.

[0111] In some embodiments of the present invention, the first joint training of the initial feature point detector and the initial feature descriptor encoder is performed based on a preset image feature point dataset, including:

[0112] Obtain the first original image in the preset image feature point dataset;

[0113] Perform transformation processing on the first original image to obtain a first transformed image;

[0114] The transformation processing includes: randomly one or more of cropping, translation, scaling, rotation, and perspective;

[0115] Input the first original image and the first transformed image into the initial feature descriptor encoder respectively to obtain a first set of feature descriptors of the first original image and a second set of feature descriptors of the first transformed image;

[0116] Calculate the first loss function according to the distances between the corresponding feature descriptors in the first set of feature descriptors and the second set of feature descriptors;

[0117] Input the first original image into the initial feature point detector to obtain a sixth feature map with several channels; each channel of the sixth feature map corresponds to an image block of the first original image; the data in each channel of the sixth feature map represents the probability value that the corresponding image block in the first original image is a feature point or the probability that the corresponding image block has no pixel points;

[0118] Input the first transformed image into an initial feature point detector to obtain seventh feature maps of several channels; each channel of the seventh feature maps corresponds to an image patch of the first transformed image; the data in each channel of the seventh feature maps represents the probability value that the corresponding image patch in the first transformed image is a feature point or the probability that the corresponding image patch has no pixel points.

[0119] Calculate a second loss function according to the ground truth of the feature points of the first original image and the sixth feature map.

[0120] Calculate a third loss function according to the ground truth of the feature points of the first transformed image and the seventh feature map.

[0121] Obtain a fourth loss function according to the first loss function, the second loss function, and the third loss function.

[0122] Optimize the preset parameters of the initial feature point detector and the initial feature descriptor encoder by using the fourth loss function to obtain a jointly trained feature point detector and feature descriptor encoder for the first time.

[0123] Specifically, take the initial feature point detector and the initial feature descriptor encoder as the initial BasePoint model. Perform the first joint training on the initial BasePoint model based on a preset image feature point dataset. Among them, the preset image feature point dataset is constructed according to a preset shape element image and the corresponding ground truth of the feature points.

[0124] Preferably, the preset shape element images include but are not limited to: basic shape element images such as line segments, triangles, rectangles, and cubes, as well as basic element images such as triangles, quadrilaterals, straight lines, cubes, checkerboards, and stars. Set the corresponding ground truth of the feature points for the pixel points in the preset shape element images, and set the ground truth of the feature points corresponding to the corner points of the preset shape element images to a preset feature value. For example, if the ground truth value of the feature points is 1 and the ground truth value of the non-feature points is 0, then set the ground truth value of the feature points corresponding to the corner points of the shape element image to 1, and the ground truth value of the other pixel points to 0. As Figure 3 shown, some example diagrams of basic element images are respectively shown, including: images of quadrilaterals (Quads), triangles (Tris), cubes (Cubes), stars (Stars), straight lines (Lines), checkerboards (Checkerboards), and quadrilateral grids (Quad Grids) and their corresponding corner points (the green dots in the figure are corner points).

[0125] Specifically, obtain a preset shape element image as the first original image I from a preset image feature point dataset; perform transformation processing on the first original image to obtain a first transformed image I _warp ; The transformation processing includes: randomly selecting one or more of cropping, translation, scaling, rotation, and perspective. Based on the transformation relationship and the feature point position information of the first original image I, the feature point position information in the first transformed image I _warp can be obtained.

[0126] Input the first original image and the first transformed image into the initial BasePoint model respectively, and use the initial feature descriptor encoder to obtain the first feature descriptor set D of the first original image and the second feature descriptor set D' of the first transformed image respectively. Denote the feature point positions of D and D' as P hw and P h’w’ .

[0127] Preferably, the transformation relationship between P hw and P h’w’ is s hwh’w’ , and the expression of s hwh’w’ is:

[0128]

[0129] where Warp() represents passing through the transformation relationship.

[0130] Specifically, the device needs to optimize the parameters of the initial feature point detector so that the Euclidean distance between the feature descriptors of the same feature points of the first original image I and the first transformed image I_warp is as small as possible, and the Euclidean distance between the feature descriptors of different feature points is as large as possible. Therefore, calculate the first loss function according to the Euclidean distance between the corresponding feature descriptors in the first feature descriptor set and the second feature descriptor set. Preferably, the first loss function adopts the hinge loss function (HingeLoss), and the expression is:

[0131]

[0132] where Lp(D, D', S) is the first loss function, λ d is a hyperparameter used to balance positive and negative samples, and the preferred value is 250; m p and m n are hyperparameters used to limit the range of loss, and the preferred values are 1.0 and 0.2 respectively., d represents the feature descriptor of the original image I, d ∈ D, d' represents the same feature descriptor d' ∈ D' in the transformed image I_warp. d T d' represents the cosine distance between the two feature descriptors d and d'. d hw represents that the feature point position in D is Phw Feature descriptor, d' h ' w ' indicates that the feature point position in D' is P h ' w 's feature descriptor.

[0133] Input the first original image I into the initial feature point detector. Use the initial feature point detector to input the first special original image I into two second connection layers and a softmax layer in sequence to obtain a sixth feature map with 65 channels, height H / 8, and width W / 8, denoted as X. The data x of 65 channels in the sixth feature map X hw , where the first 64 channels respectively correspond to the probability of each pixel point in an 8*8 image block in the first original image I being a feature point, and the remaining 1 channel represents the probability that there is no pixel point in the current image block.

[0134] Input the first transformed image I_warp into the initial feature point detector to obtain a seventh feature map with several channels; each channel of the seventh feature map corresponds to an image block of the first transformed image; the data in each channel of the seventh feature map represents the probability value that the corresponding image block in the first transformed image is a feature point or the probability that the corresponding image block has no pixel points.

[0135] According to the ground truth Y of the feature points of the first original image I and the data x of each channel in the sixth feature map hw , calculate the second loss function Lp(X, Y). Preferably, the cross-entropy loss method is used for optimization, and the expression of the second loss function is:

[0136]

[0137] where, y hw represents the ground truth of the feature points in an 8*8 image block corresponding to the original image, where the feature point is 1 and the non-feature point is 0; x hwy represents the probability value of a specific bit in x hw , y takes values in [1, 65], for example, x hw1 represents the predicted probability value of the feature point at the first position in x hw . represents the probability value that a 65-dimensional predicted vector is a feature point, and this operation is softmax.

[0138] According to the ground truth Y' of the feature points of the first transformed image I _warp and the data of each channel in the seventh feature map X', calculate the third loss function L p (X', Y').

[0139] Add the first loss function, the second loss function, and the third loss function after multiplying them by their corresponding coefficients to obtain a fourth loss function. The expression of the fourth loss function is:

[0140] L(X,X',D,D',Y,Y',S) = L p (X,Y) + L p (X',Y') + λL p (D,D',S).

[0141] Where λ is a hyperparameter used to balance the final loss, and the preferred value is 0.0001.

[0142] Optimize the preset parameters of the initial feature point detector and the initial feature descriptor encoder using the fourth loss function to obtain a first jointly trained feature point detector and feature descriptor encoder.

[0143] In some embodiments of the present invention, a second joint training is performed on the first jointly trained feature point detector and feature descriptor encoder based on an infrared image dataset, including:

[0144] Obtain a second original image in the infrared image dataset;

[0145] Perform transformation processing on the second original image to obtain a second transformed image;

[0146] The transformation processing includes: randomly selecting one or more of cropping, translation, scaling, rotation, and perspective;

[0147] Input the second original image and the second transformed image into the first jointly trained feature descriptor encoder respectively to obtain a third feature descriptor set of the second original image and a fourth feature descriptor set of the second transformed image;

[0148] Calculate a fifth loss function according to the distances between the corresponding feature descriptors in the third feature descriptor set and the fourth feature descriptor set;

[0149] Input the second original image into the first jointly trained feature point detector to obtain an eighth feature map of several channels, and determine the ground truth of the feature points of the second original image; each channel of the eighth feature map corresponds to an image patch of the second original image; the data in each channel of the eighth feature map represents the probability value that the corresponding image patch is a feature point or the probability that the corresponding image patch has no pixels;

[0150] Input the second transformed image into the initial feature point detector to obtain the ninth feature map of several channels, and determine the ground truth of the feature points of the second transformed image; each channel of the ninth feature map corresponds to an image patch of the second transformed image; each channel of the ninth feature map corresponds to an image patch of the second transformed image; the data in each channel of the ninth feature map represents the probability value that the corresponding image patch is a feature point or the probability that the corresponding image patch has no pixel points.

[0151] Calculate the sixth loss function according to the ground truth of the feature points of the second original image and the eighth feature map.

[0152] Calculate the seventh loss function according to the ground truth of the feature points of the second transformed image and the ninth feature map.

[0153] Obtain the eighth loss function according to the sixth loss function and the seventh loss function.

[0154] Optimize the preset parameters of the first jointly trained feature point detector and the first jointly trained feature descriptor encoder by using the eighth loss function to obtain the second jointly trained feature point detector and feature descriptor encoder.

[0155] Specifically, the device performs the second joint training on the first jointly trained BasePoint model based on the infrared image dataset. The infrared image dataset is constructed according to the preset infrared images. Preferably, the preset infrared images are the infrared image data of the substation equipment in the real scene.

[0156] Obtain the preset infrared image as the second original image from the infrared image dataset.

[0157] Perform a number of (e.g., 100 times) homography transformations on each second original image, such as random cropping, translation, scaling, rotation, perspective, and random superposition of basic transformations. For each second original image, obtain 100 corresponding second transformed images. Based on the transformation relationship and the feature point position information of the second original image, the feature point position information in the second transformed image can be obtained.

[0158] Input the second original image and the second transformed image into the first jointly trained BasePoint model respectively, and use the feature descriptor encoder after the first joint training to obtain the third set of feature descriptors of the second original image and the fourth set of feature descriptors of the second transformed image respectively. The device needs to optimize the parameters of the feature point detector after the first joint training so that the Euclidean distance between the feature descriptors of the same feature points in the second original image and the second transformed image is as small as possible, and the Euclidean distance between the feature descriptors of different feature points is as large as possible. Therefore, calculate the fifth loss function L according to the Euclidean distance between the corresponding feature descriptors in the third set of feature descriptors and the fourth set of feature descriptors p1 (D1, D1’, S1). The fifth loss function is the same as the first loss function, but with different input parameters

[0159] Use the feature point detector after the first joint training to extract the feature points of the second original image to obtain the eighth feature map corresponding to the second original image; and determine the true value of the feature points of the second original image by means of threshold binarization

[0160] Use the feature point detector after the first joint training to extract the feature points of the second transformed image, obtain the feature map corresponding to the second transformed image based on the transformation relationship, and add up all the feature maps corresponding to the second transformed image to obtain the ninth feature map X corresponding to the second transformed image H*W ; and determine the true value of the feature points of the second transformed image by means of threshold binarization. The value formula used is

[0161]

[0162] where x hw is the pixel of the ninth feature map X H*W . When the probability value that the image patch corresponding to the pixel of the feature map X H*W is a feature point is greater than the threshold, it is determined that the image patch corresponding to this pixel is a feature point of the second transformed image, and the true value of the feature point corresponding to this pixel is 1; when the probability value that the image patch corresponding to the pixel of the feature map X H*W is a feature point is less than the threshold, it is determined that the image patch corresponding to this pixel is a non-feature point of the second transformed image, and the true value of the feature point corresponding to this pixel is 0. The threshold is preferably set to 0.85

[0163] Calculate the sixth loss function L hw (X1, Y1) according to the true value Y1 of the feature points of the second original image and the data x p1 in each channel of the eighth feature map

[0164] Calculate the seventh loss function \(L\) according to the true values \(Y1'\) of the feature points of the second transformed image and the data of each channel in the ninth feature map \(X1'\). p1 (X1’, Y1’).

[0165] After multiplying the fifth loss function, the sixth loss function, and the seventh loss function by their corresponding coefficients and adding them together, an eighth loss function is obtained. The expression of the eighth loss function is:

[0166] L1(X1, X1', D1, D1', Y1, Y1', S1) = \(L\) p1 (X1, Y1) + \(L\) p1 (X1', Y1') + λ\(L\) p1 (D1, D1', S1).

[0167] Among them, λ is a hyperparameter used to balance the final loss, and its preferred value is 0.0001.

[0168] Use the eighth loss function to optimize the preset parameters of the first jointly trained feature point detector and the first jointly trained feature descriptor encoder, and obtain the second jointly trained feature point detector and feature descriptor encoder, denoted as the InfraredPoint model. Combine the InfraredPoint model with the backbone network to obtain the feature encoding model.

[0169] This application proposes an infrared image feature extraction method. The initial feature point detector and feature descriptor encoder are jointly trained for the first time using the basic graphic feature point data, and the feature point true values and feature descriptors of the preset infrared image are self-supervised generated by the feature point detector and feature descriptor encoder after the first joint training to construct a preset infrared image feature point dataset. Based on the preset infrared image feature point dataset, the feature point detector and feature descriptor encoder after the first joint training are jointly trained for the second time to obtain a trained feature point detector and feature descriptor encoder. The feature encoding model constructed according to the trained feature point detector and feature descriptor encoder in this application can more accurately detect the feature points and feature descriptors in the infrared image.

[0170] Figure 4 Fig. shows a schematic structural diagram of an embodiment of the infrared image feature extraction device of the present invention; as Figure 4 shown, the device 400 includes: a receiving module 410 and a feature extraction module 420;

[0171] The receiving module 410 is used to receive the target infrared image;

[0172] A feature extraction module 420, configured to input the target infrared image into a feature encoding model, extract feature points of the target infrared image by using a feature point detector, and extract descriptors of the target infrared image by using a feature descriptor encoder;

[0173] Wherein, the feature encoding model is obtained by performing a first joint training on an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point dataset, and then performing a second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset;

[0174] The preset image feature point dataset is constructed according to a preset shape element image and corresponding feature point ground truth; the infrared image dataset is constructed according to a preset infrared image.

[0175] In an alternative manner, before the feature extraction module 420 further extracts feature points of the target infrared image by using a feature point detector and extracts descriptors of the target infrared image by using a feature descriptor encoder, it further includes:

[0176] Performing a plurality of downsampling operations on the target infrared image by using a backbone network to obtain a first feature map of a target dimension;

[0177] Wherein, the backbone network includes a plurality of first connection layers, and a max pooling layer is connected between every two of the first connection layers. Each of the first connection layers includes a plurality of sub-connection layers, and each sub-connection layer includes a first convolutional layer, a first batch normalization layer, and a first non-linear activation layer connected in sequence;

[0178] Using the first feature map as the input of the feature point detector and the feature descriptor encoder respectively.

[0179] In an alternative manner, the feature point detector includes N second connection layers connected in sequence, a softmax layer, and a reshape operation module; the first (N - 1) second connection layers include a second convolutional layer, a second batch normalization layer, and a second non-linear activation layer connected in sequence, and the last second connection layer includes a second convolutional layer and a second batch normalization layer connected in sequence; N is a positive integer and N ≥ 2;

[0180] The extracting feature points of the target infrared image by using the feature point detector includes:

[0181] Inputting the first feature map into the feature point detector, and passing through a plurality of the second connection layers and the softmax layer in sequence to obtain a second feature map of a first channel number and a first dimension;

[0182] Among them, each channel of the second feature map corresponds to an image patch of the target infrared image; the data in each channel of the second feature map represents the probability value that the corresponding image patch is a feature point or the probability that the corresponding image patch has no pixel points.

[0183] Use the reshape operation module to perform a reshape operation on the second feature map of the target channel, and construct the second feature map of the target channel into a third feature map with a second number of channels and a second dimension.

[0184] Among them, each pixel point in the third feature map represents the probability value that the corresponding pixel point of the target infrared image is a feature point.

[0185] In an optional manner, the feature descriptor encoder includes M third connection layers connected in sequence, a resize operation module, and an L2 regularization operation module; the first (M - 1) third connection layers include a third convolutional layer, a third batch normalization layer, and a third non-linear activation layer connected in sequence, and the last third connection layer includes a third convolutional layer and a third batch normalization layer connected in sequence; M is a positive integer and is greater than or equal to 2.

[0186] The method for extracting the descriptor of the target infrared image by using the feature descriptor encoder includes:

[0187] Input the first feature map into the feature descriptor encoder, and after passing through several of the third connection layers, obtain a fourth feature map with a third number of channels and a third dimension.

[0188] Use the resize operation module to perform a resize operation on the fourth feature map to obtain a fifth feature map with a fourth number of channels and a fourth dimension.

[0189] Use the L2 regularization operation module to perform an L2 regularization operation on the fifth feature map to obtain a sixth feature map with a fifth number of channels and a fifth dimension.

[0190] Among them, each vector in each channel of the sixth feature map represents the descriptor of the corresponding pixel point of the target infrared image.

[0191] In an optional manner, before inputting the target infrared image into the feature encoding model, using the feature point detector to extract the feature points of the target infrared image, and using the feature descriptor encoder to extract the descriptor of the target infrared image, it further includes:

[0192] Construct an initial feature point detector and an initial feature descriptor encoder.

[0193] Perform the first joint training on the initial feature point detector and the initial feature descriptor encoder based on a preset image feature point dataset.

[0194] Based on the infrared image dataset, the first jointly trained feature point detector and feature descriptor encoder are subjected to second joint training to obtain a completely trained feature point detector and feature descriptor encoder;

[0195] The backbone network is combined with the completely trained feature point detector and feature descriptor encoder to obtain a feature encoding model.

[0196] In an optional manner, the first joint training of the initial feature point detector and the initial feature descriptor encoder based on a preset image feature point dataset includes:

[0197] Obtain the first original image in the preset image feature point dataset;

[0198] Perform transformation processing on the first original image to obtain a first transformed image;

[0199] The transformation processing includes: randomly one or more of cropping, translation, scaling, rotation, and perspective;

[0200] The first original image and the first transformed image are respectively input into the initial feature descriptor encoder to respectively obtain a first set of feature descriptors of the first original image and a second set of feature descriptors of the first transformed image;

[0201] Calculate a first loss function according to the distances between the corresponding feature descriptors in the first set of feature descriptors and the second set of feature descriptors;

[0202] The first original image is input into the initial feature point detector to obtain a sixth feature map of several channels; each channel of the sixth feature map corresponds to an image block of the first original image; the data in each channel of the sixth feature map represents the probability value that the corresponding image block in the first original image is a feature point or the probability that the corresponding image block has no pixel points;

[0203] The first transformed image is input into the initial feature point detector to obtain a seventh feature map of several channels; each channel of the seventh feature map corresponds to an image block of the first transformed image; the data in each channel of the seventh feature map represents the probability value that the corresponding image block in the first transformed image is a feature point or the probability that the corresponding image block has no pixel points;

[0204] Calculate a second loss function according to the ground truth of the feature points of the first original image and the sixth feature map;

[0205] Calculate a third loss function according to the ground truth of the feature points of the first transformed image and the seventh feature map;

[0206] Obtain a fourth loss function according to the first loss function, the second loss function, and the third loss function;

[0207] Optimize the preset parameters of the initial feature point detector and the initial feature descriptor encoder by using the fourth loss function to obtain a feature point detector and a feature descriptor encoder after the first joint training.

[0208] In an optional manner, perform second joint training on the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image dataset, including:

[0209] Obtain a second original image in the infrared image dataset;

[0210] Perform transformation processing on the second original image to obtain a second transformed image;

[0211] The transformation processing includes any one or more of random cropping, translation, scaling, rotation, and perspective;

[0212] Input the second original image and the second transformed image into the feature descriptor encoder after the first joint training respectively to obtain a third feature descriptor set of the second original image and a fourth feature descriptor set of the second transformed image;

[0213] Calculate a fifth loss function according to the distances between the corresponding feature descriptors in the third feature descriptor set and the fourth feature descriptor set;

[0214] Input the second original image into the feature point detector after the first joint training to obtain an eighth feature map of several channels, and determine the ground truth of the feature points of the second original image; each channel of the eighth feature map corresponds to an image patch of the second original image; the data in each channel of the eighth feature map represents the probability value that the corresponding image patch is a feature point or the probability that the corresponding image patch has no pixel points;

[0215] Input the second transformed image into the initial feature point detector to obtain a ninth feature map of several channels, and determine the ground truth of the feature points of the second transformed image; each channel of the ninth feature map corresponds to an image patch of the second transformed image; each channel of the ninth feature map corresponds to an image patch of the second transformed image; the data in each channel of the ninth feature map represents the probability value that the corresponding image patch is a feature point or the probability that the corresponding image patch has no pixel points;

[0216] Calculate a sixth loss function according to the ground truth of the feature points of the second original image and the eighth feature map;

[0217] Calculate a seventh loss function according to the true values of the feature points of the second transformed image and the ninth feature map;

[0218] Obtain an eighth loss function according to the sixth loss function and the seventh loss function;

[0219] Optimize the preset parameters of the first jointly trained feature point detector and the first jointly trained feature descriptor encoder by using the eighth loss function to obtain a second jointly trained feature point detector and feature descriptor encoder.

[0220] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, please refer to the partial description of the method embodiment.

[0221] This application proposes an infrared image feature extraction device that performs first joint training on an initial feature point detector and a feature descriptor encoder using basic graphic feature point data, and uses the first jointly trained feature point detector and feature descriptor encoder to self-supervisedly generate the true values of the feature points and feature descriptors of a preset infrared image to construct a preset infrared image feature point dataset. Based on the preset infrared image feature point dataset, the first jointly trained feature point detector and feature descriptor encoder are subjected to second joint training to obtain a trained feature point detector and feature descriptor encoder. The feature encoding model constructed according to the trained feature point detector and feature descriptor encoder in this application can more accurately detect the feature points and feature descriptors in the infrared image.

[0222] Figure 5 The structural schematic diagram of an embodiment of the infrared image feature extraction device of the present invention is shown. The specific implementation of the infrared image feature extraction device in the specific embodiment of the present invention is not limited.

[0223] As Figure 5 shown, the infrared image feature extraction device may include: a processor 502, a communication interface 504, a memory 506, and a communication bus 508.

[0224] Among them: the processor 502, the communication interface 504, and the memory 506 communicate with each other through the communication bus 508. The communication interface 504 is used to communicate with network elements of other devices such as clients or other servers. The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above method embodiment for infrared image feature extraction.

[0225] Specifically, the program 510 may include program code, and the program code includes computer-executable instructions.

[0226] The processor 502 may be a central processing unit (CPU), or a specific application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. One or more processors included in the infrared image feature extraction device may be of the same type, such as one or more CPUs; or they may be of different types, such as one or more CPUs and one or more ASICs.

[0227] A memory 506 is used to store a program 610. The memory 506 may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0228] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.

[0229] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present invention may be practiced without these specific details. Similarly, in order to streamline the present invention and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Among them, the claims following the specific implementation manner are hereby expressly incorporated into the specific implementation manner, where each claim itself serves as a separate embodiment of the present invention.

[0230] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive.

[0231] It should be noted that the above embodiments illustrate the present invention rather than limit the present invention, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for extracting infrared image features, characterized in that: The method comprises: Receiving target infrared images; Inputting the target infrared image into a feature coding model, extracting feature points of the target infrared image using a feature point detector, and extracting a descriptor of the target infrared image using a feature descriptor encoder; The feature coding model is obtained by first jointly training an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point data set, and then second jointly training the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image data set; The preset image feature point data set is constructed based on a preset shape element image and corresponding feature point true values; the infrared image data set is constructed based on a preset infrared image.

2. The infrared image feature extraction method according to claim 1, characterized in that: Before extracting the feature points of the target infrared image by using a feature point detector and extracting the descriptor of the target infrared image by using a feature descriptor encoder, the method further includes: Using the backbone network to perform several downsampling operations on the target infrared image to obtain a first feature map of the target dimension; The backbone network includes a plurality of first connection layers, each of which is connected by a maximum pooling layer, each of which includes a plurality of sub-connection layers, and each of which includes a first convolutional layer, a first batch normalization layer, and a first nonlinear activation layer connected in sequence; The first feature map is used as input of the feature point detector and the feature descriptor encoder respectively.

3. The infrared image feature extraction method according to claim 2, characterized in that: The feature point detector includes N second connection layers connected in sequence, a softmax layer and a shape changing operation module; the first (N-1) second connection layers include a second convolution layer, a second batch normalization layer and a second nonlinear activation layer connected in sequence, and the last second connection layer includes a second convolution layer and a second batch normalization layer connected in sequence; N is a positive integer greater than or equal to 2; The step of extracting feature points of the target infrared image by using a feature point detector comprises: Inputting the first feature map into a feature point detector, and sequentially passing through a plurality of the second connection layers and the softmax layer, to obtain a second feature map of the first dimension with the first number of channels; Wherein, each channel of the second feature map corresponds to an image block of the target infrared image; the data in each channel of the second feature map represents the probability value of the corresponding image block being a feature point or the probability that the corresponding image block has no pixel point; Using a reshape operation module to perform a reshape operation on the second feature map of the target channel, and constructing the second feature map of the target channel into a third feature map of the second dimension of the second channel number; Each pixel point in the third feature map represents a probability value of a pixel point corresponding to the target infrared image being a feature point.

4. The infrared image feature extraction method according to claim 2, characterized in that: The feature descriptor encoder includes M third connection layers connected in sequence, a resizing operation module and an L2 regularization operation module; the first (M-1) third connection layers include a third convolution layer, a third batch normalization layer and a third nonlinear activation layer connected in sequence, and the last third connection layer includes a third convolution layer and a third batch normalization layer connected in sequence; M is a positive integer greater than or equal to 2; The step of extracting the descriptor of the target infrared image by using a feature descriptor encoder comprises: Inputting the first feature map into a feature descriptor encoder, passing through a plurality of the third connection layers, to obtain a fourth feature map of a third dimension with a third number of channels; Using a resizing operation module to perform a resizing operation on the fourth feature map to obtain a fifth feature map of a fourth dimension with a fourth number of channels; Performing an L2 regularization operation on the fifth feature map using an L2 regularization operation module to obtain a sixth feature map with a fifth channel number and a fifth dimension; The vector in each channel of the sixth feature map represents a descriptor of a pixel point corresponding to the target infrared image.

5. The infrared image feature extraction method according to any one of claims 1 to 4, characterized in that: Before inputting the target infrared image into a feature coding model, extracting feature points of the target infrared image using a feature point detector, and extracting descriptors of the target infrared image using a feature descriptor encoder, the method further includes: Construct an initial feature point detector and an initial feature descriptor encoder; Performing a first joint training on an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point data set; Based on the infrared image data set, a second joint training is performed on the feature point detector and the feature descriptor encoder after the first joint training to obtain a trained feature point detector and feature descriptor encoder; The backbone network is combined with the trained feature point detector and feature descriptor encoder to obtain the feature encoding model.

6. The infrared image feature extraction method according to claim 5, characterized in that: The initial feature point detector and the initial feature descriptor encoder are first jointly trained based on a preset image feature point data set, including: Obtaining a first original image in a preset image feature point data set; Performing transformation processing on the first original image to obtain a first transformed image; The transformation processing includes: random one or more of cropping, translation, scaling, rotation and perspective; Inputting the first original image and the first transformed image into initial feature descriptor encoders respectively, to obtain a first feature descriptor set for the first original image and a second feature descriptor set for the first transformed image respectively; Calculating a first loss function according to distances between corresponding feature descriptors in the first feature descriptor set and the second feature descriptor set; Inputting the first original image into an initial feature point detector to obtain a sixth feature map of several channels; each channel of the sixth feature map corresponds to an image block of the first original image; data in each channel of the sixth feature map represents a probability value of a corresponding image block in the first original image being a feature point or a probability that the corresponding image block has no pixel point; Inputting the first transformed image into an initial feature point detector to obtain a seventh feature map of several channels; each channel of the seventh feature map corresponds to an image block of the first transformed image; data in each channel of the seventh feature map represents a probability value of the corresponding image block in the first transformed image being a feature point or a probability that the corresponding image block has no pixel point; Calculating a second loss function according to the true value of the feature point of the first original image and the sixth feature map; Calculating a third loss function according to the true values ​​of the feature points of the first transformed image and the seventh feature map; Obtaining a fourth loss function according to the first loss function, the second loss function and the third loss function; The fourth loss function is used to optimize preset parameters of the initial feature point detector and the initial feature descriptor encoder to obtain a first jointly trained feature point detector and feature descriptor encoder.

7. The infrared image feature extraction method according to claim 5, characterized in that: The feature point detector and the feature descriptor encoder after the first joint training are subjected to a second joint training based on the infrared image dataset, comprising: Acquire a second original image in the infrared image dataset; Performing transformation processing on the second original image to obtain a second transformed image; The transformation processing includes: random one or more of cropping, translation, scaling, rotation and perspective; Inputting the second original image and the second transformed image into the first jointly trained feature descriptor encoder respectively, to obtain a third feature descriptor set of the second original image and a fourth feature descriptor set of the second transformed image respectively; Calculating a fifth loss function according to the distance between the corresponding feature descriptors in the third feature descriptor set and the fourth feature descriptor set; Input the second original image into the feature point detector after the first joint training to obtain an eighth feature map of several channels, and determine the true value of the feature point of the second original image; each channel of the eighth feature map corresponds to an image block of the second original image; the data in each channel of the eighth feature map represents the probability value of the corresponding image block being a feature point or the probability that the corresponding image block has no pixel point; Input the second transformed image into an initial feature point detector to obtain a ninth feature map of several channels, and determine the true value of the feature point of the second transformed image; each channel of the ninth feature map corresponds to an image block of the second transformed image; each channel of the ninth feature map corresponds to an image block of the second transformed image; the data in each channel of the ninth feature map represents the probability value of the corresponding image block being a feature point or the probability that the corresponding image block has no pixel point; Calculating a sixth loss function according to the true values ​​of the feature points of the second original image and the eighth feature map; Calculating a seventh loss function according to the true values ​​of the feature points of the second transformed image and the ninth feature map; According to the sixth loss function and the seventh loss function, an eighth loss function is obtained; The eighth loss function is used to optimize preset parameters of the first jointly trained feature point detector and the first jointly trained feature descriptor encoder to obtain a second jointly trained feature point detector and feature descriptor encoder.

8. An infrared image feature extraction device, characterized in that: The device comprises: a receiving module and a feature extraction module; A receiving module, used for receiving an infrared image of a target; A feature extraction module, used for inputting the target infrared image into a feature coding model, extracting feature points of the target infrared image using a feature point detector, and extracting descriptors of the target infrared image using a feature descriptor encoder; The feature coding model is obtained by first jointly training an initial feature point detector and an initial feature descriptor encoder based on a preset image feature point data set, and then second jointly training the feature point detector and the feature descriptor encoder after the first joint training based on an infrared image data set; The preset image feature point data set is constructed based on a preset shape element image and corresponding feature point true values; the infrared image data set is constructed based on a preset infrared image.

9. An infrared image feature extraction device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the infrared image feature extraction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the infrared image feature extraction device / apparatus, the infrared image feature extraction device / apparatus performs the operation of the infrared image feature extraction method as described in any one of claims 1 to 7.

Images