Multi-view hyperspectral image matching method and system

Abstract

Provided are a multi-view hyperspectral image matching method and system. The multi-view hyperspectral image matching method according to an embodiment of the present invention: acquires N channel multi-view hyperspectral images; generates D channel multi-view hyperspectral images from the acquired N channel multi-view hyperspectral images; extracts feature points from the generated D channel multi-view hyperspectral images and searches for matching points; and matches the N channel multi-view hyperspectral images on the basis of the feature point matching results. Accordingly, it is possible to reduce computational complexity during hyperspectral image matching, thereby enabling real-time processing and reducing power consumption consumed in a mobile environment.

Description

Multi-point hyperspectral image alignment method and system

[0001] The present invention relates to image processing technology, and more particularly, to a method for generating a high-resolution hyperspectral image by matching hyperspectral images captured at different points in time.

[0002] Existing image alignment techniques propose methods for aligning black-and-white or color images. However, hyperspectral images, unlike typical black-and-white or color images, have a large number of channels, and the characteristics of each channel differ from those of standard images.

[0003] Therefore, if you try to align re-point hyperspectral images using existing image alignment technology, it will require a lot of computation, which will consume a lot of time and power, and may also have limitations in accurate alignment due to complexity.

[0004] Therefore, a method is required to reduce the computational complexity for re-aligning hyperspectral images.

[0005] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method for reducing the amount of computation required for image alignment by extracting meaningful information from multi-channel images acquired with a hyperspectral camera when restoring a high-resolution hyperspectral image by aligning low-resolution multi-view hyperspectral images acquired with a hyperspectral camera.

[0006] In order to achieve the above object, a multi-view hyperspectral image alignment method according to one embodiment of the present invention comprises the steps of: acquiring multi-view hyperspectral images of N channels; generating multi-view hyperspectral images of D channels from the acquired multi-view hyperspectral images of N channels; extracting feature points and searching for matching points for the generated multi-view hyperspectral images of D channels; and aligning multi-view hyperspectral images of N channels based on the feature point matching results.

[0007] The generation step may include a step of preprocessing the acquired N-channel multi-point hyperspectral images; a step of extracting D-channel multi-point hyperspectral images from the preprocessed N-channel multi-point hyperspectral images; and a step of postprocessing the extracted D-channel multi-point hyperspectral images.

[0008] The extraction step may be to extract D channels of multi-view hyperspectral images from N channels based on texture information of hyperspectral image channels.

[0009] The extraction step may include: a step of Laplacian filtering each of N channels of preprocessed multi-view hyperspectral images; a step of accumulating the absolute values ​​of pixel values ​​for each of N filtered channels for each of the multi-view hyperspectral images; and a step of selecting only D channels in order of highest accumulated values ​​for each of the multi-view hyperspectral images.

[0010] The generation step may include a step of preprocessing the acquired N-channel multi-point hyperspectral images; a step of combining the preprocessed N-channel multi-point hyperspectral images into D-channel multi-point hyperspectral images; and a step of postprocessing the combined D-channel multi-point hyperspectral images.

[0011] The combining step may include: a step of Laplacian filtering each of N channels of preprocessed multi-view hyperspectral images; a step of accumulating, for each of the multi-view hyperspectral images, the absolute values ​​of pixel values ​​for each of the N filtered channels; and a step of generating, for each of the multi-view hyperspectral images, D channel multi-view hyperspectral images by weighting the N channels according to weights determined based on the accumulated values.

[0012] The combining step may include: predicting kernels from preprocessed N-channel multi-view hyperspectral images using a deep learning network trained to predict kernels for combining D-channel multi-view hyperspectral images from N-channel multi-view hyperspectral images; and generating D-channel multi-view hyperspectral images using the predicted kernels for each of the multi-view hyperspectral images.

[0013] The search step may be to extract feature points and search for matching points for the generated D-channel multi-view hyperspectral images using a deep learning network trained to extract feature points and search for matching points for the D-channel multi-view hyperspectral images.

[0014] Preprocessing may include normalization and noise removal, and postprocessing may include replacing values ​​above or below a specific value with a specific value.

[0015] According to another aspect of the present invention, a multi-view hyperspectral image matching system is provided, comprising: a communication unit for acquiring multi-view hyperspectral images of N channels; a processor for generating multi-view hyperspectral images of D channels from the acquired multi-view hyperspectral images of N channels, extracting feature points and searching for matching points for the generated multi-view hyperspectral images of D channels, and matching the multi-view hyperspectral images of N channels based on the feature point matching results;

[0016] According to another aspect of the present invention, a multi-view hyperspectral image matching method is provided, comprising: a step of generating multi-view hyperspectral images of N channels; a step of acquiring the generated multi-view hyperspectral images of N channels; a step of generating multi-view hyperspectral images of D channels from the acquired multi-view hyperspectral images of N channels; a step of extracting feature points and searching for matching points for the generated multi-view hyperspectral images of D channels; and a step of matching the multi-view hyperspectral images of N channels based on the feature point matching results.

[0017] According to another aspect of the present invention, a hyperspectral imaging system is provided, comprising: a hyperspectral imaging camera that generates N-channel multi-view hyperspectral images; a multi-view hyperspectral image matching system that acquires N-channel multi-view hyperspectral images generated from the hyperspectral imaging camera, generates D-channel multi-view hyperspectral images from the acquired N-channel multi-view hyperspectral images, extracts feature points and searches for matching points for the generated D-channel multi-view hyperspectral images, and matches the N-channel multi-view hyperspectral images based on the feature point matching results;

[0018] As described above, according to embodiments of the present invention, since images are aligned using feature information selected from multi-channel information of multi-view hyperspectral images acquired from a hyperspectral camera, computational complexity can be reduced, real-time processing is possible, and power consumption in a mobile environment can be reduced.

[0019] Figure 1 is a multi-point hyperspectral image alignment method according to one embodiment of the present invention;

[0020] Figure 2 is a method for extracting channels from a multi-point hyperspectral image.

[0021] Figure 3 is a channel combination method of a multi-point hyperspectral image.

[0022] Figure 4 is a method for combining channels of multi-view hyperspectral images using a deep learning network.

[0023] Figure 5 is a hyperspectral imaging system according to another embodiment of the present invention;

[0024] Figure 6 is a hyperspectral image alignment system according to another embodiment of the present invention.

[0025] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0026] Conventional image alignment technology extracts feature information from multi-view images acquired from cameras, compares the extracted feature information to find matching points in different images, estimates the pose information of the camera that captured the image based on the matched points, and stitches the images together based on this.

[0027] Since existing black-and-white or color images generally have a limited number of channels, such as 1 or 3 (or 4), there are no significant limitations in performing the process described above in mobile environments or situations where computational power is limited.

[0028] In contrast, hyperspectral images typically have several to hundreds of times more channels than black-and-white or color images. Consequently, the computational complexity required for image alignment increases proportionally with the number of channels, posing limitations to real-time image alignment or in environments with limited computational resources, such as mobile and edge devices.

[0029] Accordingly, an embodiment of the present invention proposes a multi-view hyperspectral image alignment method. This technique extracts feature information that can represent channel information of hyperspectral images and performs image stitching based on the extracted feature information, thereby effectively reducing the amount of computation required for image alignment of existing hyperspectral images, thereby enabling efficient image alignment.

[0030] FIG. 1 is a diagram showing the flow of a multi-point hyperspectral image alignment method according to one embodiment of the present invention.

[0031] In order to align the multi-point hyperspectral images, first, multi-point hyperspectral images of N channels are acquired using a hyperspectral imaging camera (S110), and multi-point hyperspectral images of D channels are generated from the multi-point hyperspectral images of N channels acquired in step S110 (S120).

[0032] In step S120, for each of the re-view hyperspectral images, D channels may be sampled from N channels, or D channels may be extracted by combining N channels. In this case, D may be 2 or more or 1.

[0033] For the D channels of multi-point hyperspectral images generated in the next step S120, feature points are extracted and matching points are searched, and the N channels of multi-point hyperspectral images are aligned based on the feature point matching results (S130).

[0034] Feature extraction algorithms include Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF). Image matching methods include block matching and K-nearest neighbor (K-NN).

[0035] Below, step S120 is described in detail with reference to Fig. 2. Fig. 2 is a drawing for explaining a method for extracting channels from a multi-point hyperspectral image.

[0036] As shown, preprocessing such as pixel value normalization and denoising is performed on the N-channel multi-view hyperspectral images acquired in step S110 (S121).

[0037] Next, D-channel multi-view hyperspectral images are extracted from N-channel preprocessed multi-view hyperspectral images (S122). By selecting D channels among N channels, texture information that is distinct from each channel can be utilized.

[0038] Specifically, in order to preserve texture characteristics, for each of the re-view hyperspectral images, Laplacian filtering is performed on each of the N channels, the absolute values ​​of the pixel values ​​are accumulated for each of the N filtered channels, and then the accumulated values ​​are sorted to select only the D channels in the highest order.

[0039] Afterwards, post-processing is performed on the extracted D channel hyperspectral images, such as replacing values ​​above or below a specific value with a specific value (S123).

[0040] Meanwhile, the extraction method presented in step S122 can be replaced by another method. For example, N-channel multi-view hyperspectral images can be combined into D-channel multi-view hyperspectral images.

[0041] Specifically, as illustrated in FIG. 3, each of the N channels of the multi-view hyperspectral images is subjected to Laplacian filtering, and for each of the multi-view hyperspectral images, the absolute values ​​of the pixel values ​​for each of the N filtered channels are accumulated, and then the N channels are weighted according to the channel-specific weights determined based on the accumulated values ​​to generate multi-view hyperspectral images of D channels.

[0042] Meanwhile, in Fig. 3, D is assumed to be '1', but D may be implemented as 2 or more. For example, one channel is combined from the first and second channels, one channel is combined from the second and third channels, ..., one channel is combined from the N-1th channel and the N-1th channel.

[0043] Figure 4 illustrates a method for combining N-channel multi-view hyperspectral images with D-channel multi-view hyperspectral images using a deep learning network.

[0044] As illustrated, a deep learning network trained to predict kernels that can combine D-channel multi-view hyperspectral images from N-channel multi-view hyperspectral images is used to predict kernels from N-channel multi-view hyperspectral images, and the predicted kernels and N-channel multi-view hyperspectral images are convolved (convolved or correlated) to combine D-channel multi-view hyperspectral images.

[0045] If the kernels are implemented in the same manner as the weights presented in Fig. 3, the deep learning network that predicts the kernels in Fig. 4 can perform unsupervised learning using the N-channel multi-view hyperspectral images used in the method illustrated in Fig. 3 described above as input data and the channel-specific weights calculated from them as the correct answer data as a learning dataset.

[0046] Furthermore, it is also possible to perform feature extraction and matching on D channel multi-view hyperspectral images combined according to the method described above, using a deep learning network trained to extract feature points and search for matching points on D channel multi-view hyperspectral images.

[0047] FIG. 5 is a diagram illustrating the configuration of a hyperspectral imaging system according to another embodiment of the present invention. As illustrated, the hyperspectral imaging system according to the embodiment of the present invention is configured to include a hyperspectral camera (200) and a hyperspectral image alignment system (300).

[0048] The hyperspectral camera (200) is configured to generate multi-view hyperspectral images and can be implemented with one or more cameras. The hyperspectral image alignment system (300) aligns multi-view hyperspectral images generated by the hyperspectral camera (200) into a single hyperspectral image.

[0049] FIG. 6 is a diagram illustrating the configuration of the hyperspectral image alignment system (300) illustrated in FIG. 5. The hyperspectral image alignment system (300) according to an embodiment of the present invention is a computing system configured to include a communication unit (310), a processor (320), a storage unit (330), and a user interface (340), as illustrated, and may be implemented as a mobile device, an edge device, or the like.

[0050] The communication unit (310) is a communication interface for connection with an external network or external device, and is connected to the hyperspectral camera (200) to acquire multi-point hyperspectral images from the hyperspectral camera (200).

[0051] The processor (320) aligns the re-point hyperspectral images according to the procedures illustrated in FIGS. 1 to 4 described above. The storage unit (330) provides the storage space necessary for the processor (320) to function and operate.

[0052] The user interface (340) receives user commands, transmits them to the processor (320), and displays the results of operations performed by the processor (320). Depending on the implementation form of the hyperspectral image alignment system (300), the user interface (340) may be omitted.

[0053] So far, a preferred embodiment of a re-point hyperspectral image alignment method and system has been described in detail.

[0054] In the above embodiment, in restoring a high-resolution hyperspectral image by aligning low-resolution hyperspectral images acquired from a hyperspectral camera, the amount of computation required for image alignment can be reduced by extracting meaningful information from multi-channel images acquired with a hyperspectral camera.

[0055] In this way, since the image is aligned using feature information selected from the multi-channel hyperspectral image information acquired from the hyperspectral camera, the computational complexity can be reduced, real-time processing is possible, and power consumption in mobile environments can be reduced.

[0056] Meanwhile, it goes without saying that the technical idea according to the embodiment of the present invention can be applied to a system comprising a hyperspectral imaging camera and a hyperspectral image alignment system, in addition to the method and system for aligning multi-point hyperspectral images generated from the hyperspectral imaging camera presented in the above embodiment.

[0057] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0058] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. A step of acquiring re-point hyperspectral images of N channels; A step of generating D-channel multi-point hyperspectral images from N-channel acquired multi-point hyperspectral images; A step of extracting feature points and searching for matching points for the generated D-channel multi-point hyperspectral images; A multi-view hyperspectral image alignment method, characterized by including a step of aligning multi-view hyperspectral images of N channels based on feature point matching results.

2. In claim 1, The creation phase is, A step of preprocessing the acquired N-channel multi-point hyperspectral images; A step of extracting D-channel multi-point hyperspectral images from N-channel preprocessed multi-point hyperspectral images; A method for aligning multi-view hyperspectral images, characterized by comprising a step of post-processing multi-view hyperspectral images of extracted D channels.

3. In claim 2, The extraction step is, A multi-view hyperspectral image alignment method characterized by extracting multi-view hyperspectral images of D channels from N channels based on texture information of hyperspectral image channels.

4. In claim 2, The extraction step is, A step of Laplacian filtering each of the N channels of the preprocessed multi-point hyperspectral images; For each of the re-point hyperspectral images, a step of accumulating the absolute values ​​of pixel values ​​for each of the N filtered channels; A method for aligning multi-point hyperspectral images, characterized by including a step of selecting only D channels in order of highest accumulated value for each of the multi-point hyperspectral images.

5. In claim 2, The creation phase is, A step of preprocessing the acquired N-channel multi-point hyperspectral images; A step of combining preprocessed N-channel multi-point hyperspectral images into D-channel multi-point hyperspectral images; A method for aligning multi-view hyperspectral images, characterized by comprising the step of post-processing multi-view hyperspectral images of combined D channels.

6. In claim 5, The combination step is, A step of Laplacian filtering each of the N channels of the preprocessed multi-point hyperspectral images; For each of the re-point hyperspectral images, a step of accumulating the absolute values ​​of pixel values ​​for each of the N filtered channels; A multi-point hyperspectral image alignment method, characterized by including a step of generating multi-point hyperspectral images of D channels by weighting N channels according to weights determined based on accumulated values ​​for each of the multi-point hyperspectral images.

7. In claim 4, The combination step is, A step of predicting kernels from preprocessed N-channel multi-view hyperspectral images using a deep learning network trained to predict kernels for combining D-channel multi-view hyperspectral images from N-channel multi-view hyperspectral images; A method for aligning multi-view hyperspectral images, characterized by comprising the step of generating multi-view hyperspectral images of D channels using predicted kernels for each of the multi-view hyperspectral images.

8. In claim 7, The exploration phase is: A multi-view hyperspectral image alignment method characterized by extracting feature points and searching for matching points for generated D-channel multi-view hyperspectral images using a deep learning network trained to extract feature points and search for matching points for D-channel multi-view hyperspectral images.

9. In claim 2, Preprocessing is, Includes normalization and noise removal, Post-processing is, A multi-point hyperspectral image alignment method characterized by replacing values ​​above or below a specific value with a specific value.

10. A communication unit for acquiring re-point hyperspectral images of N channels; A multi-view hyperspectral image alignment system, characterized by including a processor for generating D-channel multi-view hyperspectral images from N-channel acquired multi-view hyperspectral images, extracting feature points and searching for matching points for the generated D-channel multi-view hyperspectral images, and aligning N-channel multi-view hyperspectral images based on the feature point matching results.

11. A step of generating re-point hyperspectral images of N channels; A step of acquiring re-point hyperspectral images of the generated N channels; A step of generating D-channel multi-point hyperspectral images from N-channel acquired multi-point hyperspectral images; A step of extracting feature points and searching for matching points for the generated D-channel multi-point hyperspectral images; A multi-view hyperspectral image alignment method, characterized by including a step of aligning multi-view hyperspectral images of N channels based on feature point matching results.

12. A hyperspectral imaging camera that produces multi-point hyperspectral images of N channels; A hyperspectral imaging system, characterized by including a multi-view hyperspectral image alignment system which acquires N channels of multi-view hyperspectral images generated from a hyperspectral imaging camera, generates D channels of multi-view hyperspectral images from the acquired N channels of multi-view hyperspectral images, extracts feature points and searches for matching points for the generated D channels of multi-view hyperspectral images, and aligns the N channels of multi-view hyperspectral images based on the feature point matching results.

Images