Non-local information matching acceleration method and device

Abstract

The present invention proposes a non-local information matching acceleration method and device, wherein the method includes: obtaining an image to be matched, using a feature detection algorithm to obtain a feature map of the image to be matched; using a feature point detection method to extract feature point information from the feature map, extracting a sample block of the image to be matched and a feature sample block of the feature map according to the position of the feature point, wherein the sample block is an image block centered on the position of the feature point in the image to be matched, and the feature sample block is an image block centered on the position of the feature point in the feature map; using a neural network convolution layer, using the feature sample block as a convolution kernel to convolve the feature image, and calculating a global matching heat map, the global matching heat map characterizes the similarity between the sample block and each image block in the image to be matched. The present invention uses a neural network convolution layer to perform global matching of image features, accelerates the matching speed, and can perform multi-image and multi-channel parallel operations, effectively improving the speed and accuracy of non-local matching in the image processing system.

Description

Technical Field

[0001] The invention belongs to the field of computer vision. Background Art

[0002] In the field of computer vision, image reconstruction based on non-local algorithms is widely used. Image reconstruction based on non-local algorithms, that is, using the characteristics of a large number of similar blocks in the image, reconstructs the area around the target pixel based on the similar blocks. In 2005, Antoni Buades and others from the University of the Balearic Islands developed a non-local information matching method using the information encoded in the entire image. When reconstructing each pixel, the algorithm first calculates the similarity between a fixed neighborhood centered on the pixel and other neighborhoods in the entire image. The similarity is to calculate the Euclidean distance between the pixel values ​​of each point in the neighborhood. The smaller the distance, the more similar the two neighborhoods are. Then the similarity is used as a weight to reconstruct the pixel. This method has achieved remarkable results in the field of computer vision and is a key step in image processing, but it also has obvious disadvantages: pixel-level image block matching is inefficient and time-consuming.

[0003] To solve the problem of low matching efficiency, Jin Wang et al. from Zhejiang University proposed a method for quickly comparing neighborhood similarities, which uses the sum square image (SSI) and fast Fourier transform (FFT) to convert pixel-by-pixel image block matching into SSI pre-calculation and efficient FFT. This fast non-local matching algorithm is about 50 times faster than the original non-local algorithm, and its reconstruction results are close to those of traditional matching methods.

[0004] In terms of matching accuracy, Liheng Bian et al. from Beijing Institute of Technology believe that using Euclidean distance to measure the similarity between different image blocks will lose fine structural details and reduce the reconstruction effect. Because these indicators are global strength evaluations, image block matching for each fixed number of iterations may lead to repeated iterations or insufficient information utilization at different iteration stages, resulting in insufficient matching accuracy. Based on this basis, they used the Structural Similarity Index (SSIM) to better measure similarity, because SSIM uses brightness differences, contrast and structural differences to define overall indicators. The comprehensive evaluation is not only more similar to human vision, but also more consistent with the subsequent low-rank regularization that requires structural similarity, which better improves the matching accuracy. The disadvantage of non-local information matching based on SSIM is that it takes longer than traditional matching methods. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, a first object of the present invention is to propose a non-local information matching acceleration method for improving the speed and accuracy of non-local matching in an image processing system.

[0007] The second objective of the present invention is to provide a non-local information matching acceleration device.

[0008] A third object of the present invention is to provide a computer device.

[0009] A fourth object of the present invention is to provide a computer-readable storage medium.

[0010] To achieve the above object, a first embodiment of the present invention proposes a non-local information matching acceleration method, comprising:

[0011] Acquire an image to be matched, and obtain a feature map of the image to be matched using a feature detection algorithm;

[0012] Using a feature point detection method, feature point information is extracted from the feature map, and a sample block of the image to be matched and a feature sample block of the feature map are extracted according to the position of the feature point, wherein the sample block is an image block centered at the position of the feature point in the image to be matched, and the feature sample block is an image block centered at the position of the feature point in the feature map;

[0013] Using a neural network convolution layer, the feature sample block is used as a convolution kernel to convolve the feature image, and a global matching heat map is calculated;

[0014] The similarity between the sample block and each image block in the image to be matched is determined according to the global matching heat map.

[0015] In addition, the non-local information matching acceleration method according to the above embodiment of the present invention may also have the following additional technical features:

[0016] Further, in one embodiment of the present invention, the convolving the feature image with the feature sample block as a convolution kernel includes:

[0017] Multiple feature sample blocks are used as multi-channel convolution kernels to perform synchronous convolution on the feature image.

[0018] Furthermore, in one embodiment of the present invention, it also includes:

[0019] Each sample block corresponds to a global matching heat map, and each pixel value in the global matching heat map represents the similarity between the sample block and each image block in the image.

[0020] Furthermore, in one embodiment of the present invention, after determining the similarity between the sample block and each image block in the image to be matched according to the global matching heat map, the method further includes:

[0021] The heat value of each pixel point in the global matching heat map is sorted, similar image blocks are obtained, and non-local information matching is completed.

[0022] Furthermore, in one embodiment of the present invention, the step of sorting the heat value of each pixel of the global matching heat map to obtain similar image blocks includes:

[0023] Sort the heat value of each pixel in the global matching heat map by size, and obtain similar image blocks based on the coordinates.

[0024] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a non-local information matching acceleration device, comprising the following modules:

[0025] An acquisition module is used to acquire an image to be matched and obtain a feature map of the image to be matched using a feature detection algorithm;

[0026] An extraction module, used to extract feature point information from the feature map using a feature point detection method, and extract a sample block of the image to be matched and a feature sample block of the feature map according to the position of the feature point, wherein the sample block is an image block centered on the position of the feature point in the image to be matched, and the feature sample block is an image block centered on the position of the feature point in the feature map;

[0027] A calculation module, used to use a neural network convolution layer to convolve the feature image with the feature sample block as a convolution kernel, and calculate a global matching heat map;

[0028] A matching module is used to determine the similarity between the sample block and each image block in the image to be matched according to the global matching heat map.

[0029] Furthermore, in one embodiment of the present invention, the calculation module is further used for:

[0030] Multiple feature sample blocks are used as multi-channel convolution kernels to perform synchronous convolution on the feature image.

[0031] Furthermore, in one embodiment of the present invention, the matching module further includes:

[0032] The matching unit is used to sort the thermal values ​​of each pixel point of the global matching thermal map, obtain similar image blocks, and complete non-local information matching.

[0033] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the non-local information matching acceleration method as described above.

[0034] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the non-local information matching acceleration method as described above is implemented.

[0035] The non-local information matching acceleration method and device proposed in the embodiment of the present invention aims at the shortcomings of the existing non-local information matching methods, which are slow and time-consuming. The image blocks where the feature points of the feature images are located are used as feature sample blocks, and the convolution operation is used to perform global matching of image features, and finally obtain the non-local information matching results. The present invention takes into account the global structural information of the image, and benefits from the acceleration of the neural network, and can perform multi-image multi-channel parallel operations, effectively improving the speed and accuracy of non-local information matching in the image processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A flowchart of a non-local information matching acceleration method provided in an embodiment of the present invention.

[0038] Figure 2 This is a feature extraction and feature point detection diagram provided by an embodiment of the present invention.

[0039] Figure 3 A schematic diagram of a matching method and a global matching heat map generation method provided in an embodiment of the present invention.

[0040] Figure 4 A schematic diagram of a flow chart of a non-local information matching acceleration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0042] The following describes the non-local information matching acceleration method and device according to the embodiments of the present invention with reference to the accompanying drawings.

[0043] Example 1

[0044] Figure 1 A flowchart of a non-local information matching acceleration method provided in an embodiment of the present invention.

[0045] like Figure 1As shown, the non-local information matching acceleration method includes the following steps:

[0046] S101: Acquire an image to be matched, and obtain a feature map of the image to be matched using a feature detection algorithm;

[0047] S102: extracting feature point information from the feature map using a feature point detection method, and extracting a sample block of the image to be matched and a feature sample block of the feature map according to the position of the feature point, wherein the sample block is an image block centered at the position of the feature point in the image to be matched, and the feature sample block is an image block centered at the position of the feature point in the feature map;

[0048] S103: using the convolution layer of the neural network, convolving the sample block with the feature image as a convolution kernel, and calculating a global matching heat map;

[0049] S104: Determine the similarity between the sample block and each image block in the image to be matched according to the global matching heat map.

[0050] Furthermore, in one embodiment of the present invention, the feature sample block is used as a convolution kernel to convolve the feature image, including:

[0051] Multiple feature sample blocks are used as multi-channel convolution kernels to perform synchronous convolution on the feature image.

[0052] Furthermore, in one embodiment of the present invention, it also includes:

[0053] Each sample block corresponds to a global matching heat map, and the heat value of each pixel in the global matching heat map represents the similarity between the sample block and each image block in the image.

[0054] Furthermore, in one embodiment of the present invention, after determining the similarity between the sample block and each image block in the image according to the global matching heat map, the method further includes:

[0055] Sort the heat values ​​of each pixel in the global matching heat map, obtain similar image blocks, and complete non-local information matching.

[0056] Furthermore, in one embodiment of the present invention, the heat value of each pixel of the global matching heat map is sorted to obtain similar image blocks, including:

[0057] The thermal values ​​of each pixel in the global matching thermal map are sorted by size to obtain a set number of points with the highest thermal values, and similar image blocks are obtained based on the coordinates of these points.

[0058] Example 2

[0059] like Figure 2As shown in the figure, firstly, the edge detection algorithm is used as the feature detection algorithm, and the corner point extraction algorithm is used as the feature point detection algorithm. The image is edge detected to obtain a binary edge image. Based on the edge image, the edge map is corner detected using the corner point detection method. The m corner point coordinates extracted have rich details in the area where the corner points are located. Select m1 (m1≤m) corner point coordinates, and extract m1 a×a image sub-blocks with the corner point coordinates as the center point, which are m1 feature sample blocks. The parameter m1 can be used to control the number of feature sample blocks.

[0060] like Figure 3 As shown in the figure, the second step is to obtain the global matching heat map of the feature sample block for the entire image through convolution operation. The m1 feature sample blocks are used as convolution kernels to convolve with the edge map to obtain m1 global matching heat maps. The global matching heat map reflects the structural similarity between the sample block and the image to be matched. The value of each pixel in the global matching heat map represents the size of the thermal value. The thermal value represents the similarity between the sample block and all a×a image blocks in the image to be matched. The coordinates of the thermal value in the global matching heat map correspond to the coordinates of each image block. The thermal values ​​are sorted by size, and the coordinates corresponding to the first n thermal values ​​are output, and finally n similar blocks of the sample block are obtained. The convolution operation of the feature sample block on the edge map is realized through the convolution layer of the neural network, which can perform multi-image multi-channel global matching, and benefit from the acceleration of neural network calculation, it can complete non-local information matching at a faster speed and higher accuracy.

[0061] The non-local information matching acceleration method and device proposed in the embodiment of the present invention aims at the shortcomings of the existing non-local information matching methods, which are slow and time-consuming. The image blocks where the feature points of the feature images are located are used as feature sample blocks, and the convolution operation is used to perform global matching of image features, and finally obtain the non-local information matching results. The present invention takes into account the global structural information of the image, and benefits from the acceleration of the neural network, and can perform multi-image multi-channel parallel operations, effectively improving the speed and accuracy of non-local information matching in the image processing system.

[0062] In order to implement the above embodiment, the present invention also proposes a non-local information matching acceleration device.

[0063] Figure 4 A schematic diagram of the structure of a non-local information matching acceleration device provided in an embodiment of the present invention.

[0064] like Figure 4 As shown, the non-local information matching acceleration device includes: an acquisition module 100, an extraction module 200, a calculation module 300, and a matching module 400;

[0065] The acquisition module is used to acquire the image to be matched and obtain the feature map of the image to be matched by using the feature detection algorithm;

[0066] An extraction module is used to extract feature point information from a feature map using a feature point detection method, and extract a sample block of the image to be matched and a feature sample block of the feature map according to the position of the feature point, wherein the sample block is an image block centered at the position of the feature point in the image to be matched, and the feature sample block is an image block centered at the position of the feature point in the feature map;

[0067] A calculation module is used to use the convolution layer of the neural network to convolve the feature image with the feature sample block as a convolution kernel to calculate a global matching heat map;

[0068] The matching module is used to determine the similarity between the sample block and each image block in the image to be matched according to the global matching heat map.

[0069] Furthermore, in one embodiment of the present invention, the computing module is further configured to:

[0070] Multiple feature sample blocks are used as multi-channel convolution kernels to perform synchronous convolution on the feature image.

[0071] Furthermore, in one embodiment of the present invention, the matching module further includes:

[0072] The matching unit is used to sort the heat value of each pixel in the global matching heat map, obtain similar image blocks, and complete non-local information matching.

[0073] To achieve the above-mentioned purpose, a third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the multispectral sensor calibration method as described above is implemented.

[0074] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the multi-spectral sensor calibration method as described above is implemented.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0076] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0077] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A non-local information matching acceleration method, It is characterized in that The following steps are involved: Acquire an image to be matched, and obtain a feature map of the image to be matched using a feature detection algorithm; Using a feature point detection method, feature point information is extracted from the feature map, and a sample block of the image to be matched and a feature sample block of the feature map are extracted according to the position of the feature point, wherein the sample block is an image block centered at the position of the feature point in the image to be matched, and the feature sample block is an image block centered at the position of the feature point in the feature map; Using the convolution layer of the neural network, the feature sample block is used as a convolution kernel convolution feature map to calculate a global matching heat map; The similarity between the sample block and each image block in the image to be matched is determined according to the global matching heat map, wherein each sample block corresponds to a global matching heat map, and each pixel value in the global matching heat map represents the similarity between the sample block and each image block in the image to be matched.

2. The method according to claim 1, It is characterized in that The method of using the feature sample block as a convolution kernel convolution feature map includes: Multiple feature sample blocks are used as multi-channel convolution kernels to perform synchronous convolution on the feature map.

3. The method according to claim 1, It is characterized in that After determining the similarity between the sample block and each image block in the to-be-matched image according to the global matching heat map, the method further includes: The heat value of each pixel point in the global matching heat map is sorted, similar image blocks are obtained, and non-local information matching is completed.

4. The method according to claim 3, It is characterized in that The step of sorting the heat value of each pixel point of the global matching heat map to obtain similar image blocks includes: Sort the heat values ​​of each pixel in the global matching heat map by size, and obtain similar image blocks based on the coordinates.

5. A non-local information matching acceleration device, It is characterized in that Includes the following modules: An acquisition module is used to acquire an image to be matched and obtain a feature map of the image to be matched using a feature detection algorithm; An extraction module, used to extract feature point information from the feature map using a feature point detection method, and extract a sample block of the image to be matched and a feature sample block of the feature map according to the position of the feature point, wherein the sample block is an image block centered on the feature point position in the image to be matched, and the feature sample block is an image block centered on the feature point position in the feature map; A calculation module is used to use the convolution layer of the neural network to convolve the feature sample block as the convolution kernel feature map and calculate the global matching heat map; A matching module is used to determine the similarity between the sample block and each image block in the image to be matched according to the global matching heat map, wherein each sample block corresponds to a global matching heat map, and each pixel value in the global matching heat map represents the similarity between the sample block and each image block in the image to be matched.

6. The device according to claim 5, It is characterized in that The computing module is further used for: Multiple feature sample blocks are used as multi-channel convolution kernels to perform synchronous convolution on the feature map.

7. The device according to claim 5, It is characterized in that The matching module further includes: The matching unit is used to sort the thermal values ​​of each pixel point of the global matching thermal map, obtain similar image blocks, and complete non-local information matching.

8. A computer device, It is characterized in that The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the non-local information matching acceleration method as described in any one of claims 1 to 4 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the non-local information matching acceleration method as described in any one of claims 1 to 4 is implemented.

Images