Hyperspectral image processing method and apparatus, electronic device, and medium

Abstract

The application provides a hyperspectral image processing method and device, electronic equipment and medium, the hyperspectral image processing method comprises the following steps: according to the hyperspectral image processing request, the first hyperspectral image is acquired, and the projection invariant of the first hyperspectral image is extracted;According to the projection invariant, a homography matrix is constructed;A rectangular network is constructed for the first hyperspectral image, an energy function is constructed according to the homography matrix and the projection invariant, the first hyperspectral image is matched through the energy function, and the second hyperspectral image is obtained;The second hyperspectral image is fused and processed by using a VIT deep learning fusion network, and the third hyperspectral image is obtained. The image registration and fusion accuracy are improved by the point-line matching and deep learning method, the space hyperspectral images can be effectively fused, and the target saliency of space images is improved.

Description

Technical Field

[0001] This invention relates to the field of computer remote sensing data processing technology, and in particular to a hyperspectral image processing method, apparatus, electronic device, and medium. Background Technology

[0002] The development of remote sensing technology has provided an effective technical means for humankind to understand its environment and utilize natural resources. Remote sensing sensors are diverse, and different sensors produce different imaging characteristics for the same scene, generating multi-source remote sensing images such as hyperspectral images and other imagery. Among these, spectral detection technology relies on the different spectra of objects, which arise from differences in the object's surface and internal structure. For example, grasslands, trees, and snow all have specific spectral curves, exhibiting clear differences that spectral detection technology can quickly distinguish. However, spectral detection often encounters situations where "different objects share the same spectrum." While remote sensing cannot accurately detect low-contrast targets, it has unique advantages in identifying targets that are close to the background but of different materials.

[0003] To efficiently and comprehensively process and utilize hyperspectral image data, it is necessary to register and fuse multi-source remote sensing images of the same scene. Multi-source remote sensing image fusion utilizes specific techniques to remove redundant information from images taken from different sources of the same scene, while combining complementary information to generate a clearer, more accurate, and more comprehensive image describing the scene. It has wide applications in both military and civilian fields.

[0004] Traditional remote sensing image matching algorithms have low matching accuracy. Different matching and fusion rules need to be manually formulated for different types of remote sensing images. Furthermore, the fusion quality is closely related to the image decomposition method, the decomposition level, and the fusion rules selected at each level, resulting in inconsistent fusion effects from the algorithms. Summary of the Invention

[0005] The main objective of this invention is to propose a hyperspectral image processing method, apparatus, electronic device, and medium that improves image registration and fusion accuracy through point-line matching and deep learning, effectively fusing aerospace hyperspectral images and improving the target salience of aerospace images.

[0006] One aspect of the present invention provides a hyperspectral image processing method, comprising:

[0007] According to the hyperspectral image processing request, a first hyperspectral image is acquired, and the projection invariants of the first hyperspectral image are extracted. The projection invariants are used to characterize the registration points and registration lines of the first hyperspectral image.

[0008] Construct a homography matrix based on the projection invariants;

[0009] A rectangular network is constructed for the first hyperspectral image, and an energy function is constructed based on the homography matrix and the projection invariant. The first hyperspectral image is then matched using the energy function to obtain a second hyperspectral image, which is used to characterize the coarse registration image.

[0010] The second hyperspectral image is fused using a VIT deep learning fusion network to obtain a third hyperspectral image, which is used to characterize a high-precision image.

[0011] According to the hyperspectral image processing method, extracting the projection invariants of the first hyperspectral image includes:

[0012] The registration points were searched from the first hyperspectral image using the SURF algorithm;

[0013] The LSD algorithm is used to obtain registration line pairs from the first hyperspectral image. Sub-regions are divided according to the registration line pairs. The sub-regions are divided into original line segments obtained by the LSD algorithm. The neighborhood of the original line segments is divided into left and right neighborhoods according to the gradient direction. When any pixel in the neighborhood of the line of each line segment has a distance from the line and a distance from the vertical bisector that satisfies a preset value, the line is used as the registration line.

[0014] According to the hyperspectral image processing method described above, the method further includes:

[0015] The number of features of the projection invariants is set, point-line invariants are constructed, and coplanar subregions are matched using point-line invariants to find the registration points and registration lines, thereby obtaining multiple projection invariants.

[0016] According to the hyperspectral image processing method, constructing a homography matrix based on the projection invariants includes:

[0017] The RANSAC method is used to refine the registration points, and a homography matrix vector variable is constructed. While keeping the homography matrix vector variable to a minimum, the homography matrix of the transformation between the images to be matched is calculated, and the homography matrix is ​​calculated and its minimum value is obtained by using the SVD algorithm. The images to be matched are the first hyperspectral images corresponding to the registration points and the registration lines.

[0018] According to the hyperspectral image processing method, a rectangular network is constructed for the first hyperspectral image, an energy function is constructed based on the homography matrix and the projection invariants, and the first hyperspectral image is matched using the energy function to obtain a second hyperspectral image, including:

[0019] A rectangular network is constructed for the image to be stitched, the rectangular network having grid vertex coordinate indices and original image grid vertex coordinates;

[0020] Using the homography matrix, an energy function is constructed. The vertex coordinates of the distorted image grid are calculated while maintaining the lowest energy level in the energy function. A grid transformation is then used to obtain the distorted corrected image to be stitched. The stitching formula is as follows:

[0021]

[0022] in, The coordinates of the original image grid vertices. These are the coordinates of the distorted mesh vertices. To construct the energy function, This is a protection item for local and global lines. Used to control the overlap of matching point pairs and line pairs after splicing. The second hyperspectral image is obtained by maintaining the slope of the grid lines and uniform adjacent grids to control distortion during the stitching process.

[0023] According to the hyperspectral image processing method, the second hyperspectral image is fused using a VIT deep learning fusion network to obtain a third hyperspectral image, including:

[0024] A VIT-based deep learning fusion network is created using the second hyperspectral image as input;

[0025] Image features at different scales of the second hyperspectral image are extracted by the VIT deep learning fusion network, and a fusion feature map is obtained. The fusion hyperspectral image is then reconstructed by self-decoding the fusion feature map.

[0026] The fused hyperspectral image is processed using VIT-based deep learning to obtain the third hyperspectral image.

[0027] According to the hyperspectral image processing method described above, the VIT deep learning fusion network further includes:

[0028] When obtaining the fused feature map, global features are preserved through a loss function, which is:

[0029] ,

[0030] in, For coefficients, pixels The similarity loss function is

[0031] ,

[0032] in, For coefficients, Structural similarity loss function

[0033] ,

[0034] SSIM is used to represent the structural similarity between two images. For the merged image, This is the original image.

[0035] The technical solution of the present invention also includes a hyperspectral image processing device, comprising:

[0036] The first module is used to acquire a first hyperspectral image according to a hyperspectral image processing request, and extract the projection invariants of the first hyperspectral image, wherein the projection invariants are used to characterize the registration points and registration lines of the first hyperspectral image.

[0037] The second module is used to construct a homography matrix based on the projection invariants;

[0038] The third module is used to construct a rectangular network for the first hyperspectral image, construct an energy function based on the homography matrix and the projection invariant, and match the first hyperspectral image through the energy function to obtain a second hyperspectral image, which is used to characterize the coarse registration image.

[0039] The fourth module is used to perform fusion processing on the second hyperspectral image using a VIT deep learning fusion network to obtain a third hyperspectral image, which is used to characterize a high-precision image.

[0040] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0041] The memory is used to store programs;

[0042] The processor executes the program to implement the medical user information management method described above.

[0043] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0046] Figure 1 This is a schematic flowchart of the hyperspectral image processing method according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the VIT deep learning fusion network processing flow according to an embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of the VIT deep learning fusion network structure according to an embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of a hyperspectral image processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0050] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0051] refer to Figure 1 ,in Figure 1 This is a schematic flowchart of a hyperspectral image processing method according to an embodiment of the present invention, which includes, but is not limited to, steps S100 to S400:

[0052] S100: According to the hyperspectral image processing request, acquire the first hyperspectral image and extract the projection invariants of the first hyperspectral image. The projection invariants are used to characterize the registration points and registration lines of the first hyperspectral image.

[0053] In some embodiments, the SURF algorithm is used to search for registration points from the first hyperspectral image; the LSD algorithm is used to obtain registration line pairs from the first hyperspectral image; and secondary region division is performed based on the registration line pairs. The secondary region division includes the original line segments obtained by the LSD algorithm. The neighborhood of the original line segments is divided into left and right neighborhoods according to the gradient direction. When any pixel's distance from the line and its distance from the vertical bisector in the neighborhood of the line of each line segment meet a preset value, the line is used as the registration line.

[0054] In some embodiments, the number of features of the projection invariants is set, point-line invariants are constructed, and coplanar sub-regions are matched using point-line invariants to find registration points and registration lines, thereby obtaining multiple projection invariants.

[0055] S200, construct the homography matrix based on the projection invariants.

[0056] In some embodiments, the RANSAC method is used to refine the registration points and construct homography matrix vector variables. While keeping the homography matrix vector variables to a minimum, the homography matrix of the transformation between the images to be matched is calculated, and the homography matrix is ​​calculated and its minimum value is obtained by the SVD algorithm. The images to be matched are the first hyperspectral images corresponding to the registration points and registration lines.

[0057] S300: Construct a rectangular network for the first hyperspectral image, construct an energy function based on the homography matrix and projection invariants, and match the first hyperspectral image using the energy function to obtain a second hyperspectral image. The second hyperspectral image is used to characterize the coarse registration image.

[0058] In some embodiments, a rectangular network is constructed for the images to be stitched, and the rectangular network has grid vertex coordinate indices and original image grid vertex coordinates;

[0059] An energy function is constructed using the homography matrix. The vertex coordinates of the distorted image grid are calculated while maintaining the lowest energy level in the energy function. The grid transformation is then used to obtain the distorted corrected image to be stitched. The stitching formula is as follows:

[0060]

[0061] in, The coordinates of the original image grid vertices. These are the coordinates of the distorted mesh vertices. To construct the energy function, This is a protection item for local and global lines. Used to control the overlap of matching point pairs and line pairs after splicing. By maintaining the slope of the grid lines and ensuring uniformity between adjacent grids to control distortion during the stitching process, a second hyperspectral image is obtained.

[0062] S400 uses the VIT deep learning fusion network to fuse the second hyperspectral image to obtain the third hyperspectral image, which is used to characterize the high-precision image.

[0063] In some embodiments, a VIT-based deep learning fusion network is created with a second hyperspectral image as input; image features at different scales of the second hyperspectral image are extracted through the VIT deep learning fusion network, and a fusion feature map is obtained; the fusion feature map is reconstructed by self-decoding; and the fusion hyperspectral image is processed by VIT-based deep learning to obtain a third hyperspectral image.

[0064] In some embodiments, reference Figure 2 The diagram shown illustrates the processing flow of the VIT deep learning fusion network, which includes:

[0065] This embodiment uses a point-line registration algorithm to coarsely align the acquired hyperspectral images, and then constructs a VIT deep learning fusion network to perform fine registration and fusion of the coarsely aligned hyperspectral images and information.

[0066] Step 1: Coarse registration of hyperspectral images based on point-line matching algorithm

[0067] In the process of image matching, in order to preserve the local and global geometric structure of wide parallax images while reducing artifacts and distortion, a point-line matching image processing method is adopted to achieve coarse registration of hyperspectral images, thus preparing for subsequent image fusion.

[0068] (1) The SURF algorithm and LSD algorithm are used to search for hyperspectral images to find the registration points and registration lines of the two images respectively;

[0069] (2) By defining the projection invariant feature number, point and line invariants are constructed, coplanar subregions are matched, and more matching points and matching lines are obtained;

[0070] (3) The RANSAC method is used to refine the registration points, construct the homography matrix vector variable, and obtain the homography matrix for hyperspectral image matching;

[0071] (4) Construct a rectangular network for the hyperspectral image, and construct an energy function using the matched point pairs, line pairs and homography matrix to ensure that the hyperspectral image is matched with the lowest energy function.

[0072] For example, a specific implementation method for coarse registration of hyperspectral images based on a point-line matching algorithm includes:

[0073] (1) The SURF algorithm is used to search for registration point pairs in the hyperspectral image. ,in This indicates the number of matching point pairs. The LSD algorithm is used to obtain the registration line pairs of the hyperspectral image. ,in The number of matching line pairs is described, and sub-region partitioning is achieved based on line detection: lines can be considered as the intersection of planes, so it is assumed that the neighborhood determined by the line segment length can be regarded as a locally coplanar sub-region of the image. The original line segments are obtained using LSD, and then the neighborhood of each line segment is divided into left and right neighborhoods based on the gradient direction (because points on different sides of a line may not be coplanar). The gradient of a line is defined as the average gradient of all points on it. Within the neighborhood of a line, any pixel's distance to the line is less than... The distance to the perpendicular bisector is less than ,in and These are parameters used to divide line segments, and they can be adjusted.

[0074] (2) Define the projection invariant feature number (CN) to construct point and line invariants, and match more points and lines in the coplanar subregion.

[0075] For reference hyperspectral region The projection space, on which This includes the matching points and matching line endpoints in steps (1) and (2).

[0076] dot set ,

[0077] dot set ,

[0078] For line segments

[0079]

[0080] Set the intersection points of any two line segments. The endpoints defined above can be used and Represented as:

[0081] (1)

[0082] in and The scaling factor, the eigenvalue of the projection invariant, is calculated as follows:

[0083] (2)

[0084] Similarly, in the images to be merged, The set of matching points is The corresponding point set exists.

[0085] ,

[0086] Projection Invariant Eigenvalues .

[0087] The projection invariant eigenvalues ​​of the point set in the overlapping region of a hyperspectral image should remain unchanged, and

[0088] ,

[0089] This allows for the selection of more matching points. and lines

[0090] .

[0091] (1) The RANSAC method is used to refine the registration points and construct the homography matrix vector variables. In maintaining Under the condition of minimization, the homography matrix H of the transformation between the images to be matched is obtained, and the matrix (formula) is minimized using the SVD algorithm: .

[0092] (3)

[0093] Among them, the use of Indicates the starting and ending points of the line. Indicates the endpoint of the line.

[0094] (2) Construct a rectangular network for the image to be stitched together, with the grid vertex coordinate indices as follows: Using vectors This represents the vertex coordinates of the original image grid.

[0095] (3) Construct the energy function using the homography matrix H. The distorted mesh vertex coordinates are obtained while maintaining the lowest energy level.

[0096] ,

[0097] The distortion-corrected image img1_wrap of the image to be stitched is obtained using mesh transformation.

[0098] (4)

[0099] in This provides protection for local and global straight lines, addressing the issue of preserving linear structures. Control the overlap of matched point pairs and line pairs after splicing. Distortion during the splicing process is controlled by maintaining the slope of the grid lines and ensuring uniformity between adjacent grids.

[0100] Step 2: Hyperspectral image fusion based on VIT deep learning fusion network.

[0101] A deep learning fusion network based on VIT fuses hyperspectral images, combining information from different bands to obtain significant fusion components. Compared to CNN-based fusion algorithms, IFT has superior long feature extraction capabilities for hyperspectral image fusion. This invention utilizes a hyperspectral fusion network structure built on Fusion VIT to achieve hyperspectral image fusion.

[0102] (1) Construct a hyperspectral image and information fusion network structure including three modules: Encoder Block, Decoder Block and Spatio-VIT Block;

[0103] (2) Based on the requirements of hyperspectral image fusion, a specific loss function is selected to preserve the global features of the fusion process;

[0104] (3) Input the coarsely registered hyperspectral image, extract image features of different scales through an autoencoder, obtain the fusion feature map of the m-th layer, and reconstruct the fused hyperspectral image through self-decoding; train the Spatio-VIT Block to obtain a high-precision hyperspectral fusion image.

[0105] Specific implementations of hyperspectral image fusion based on VIT deep learning fusion networks include:

[0106] (1) Constructing such Figure 3 The diagram shows a hyperspectral image fusion network structure consisting of three modules: Encoder Block, Decoder Block, and Spatio-VITBlock. The Encoder-Decoder module comprises multiple... The convolutional layers are composed of ReLU

[48] , with the activation function being ReLU

[48] and the number of input channels being . The output channel is Input hyperspectral image Image features at different scales were extracted using an autoencoder. , After undergoing corresponding axial attention and spatial feature calculations, the fused feature map of the m-th layer is obtained using Add Block, i.e.

[0107] (5)

[0108] (2) Based on the requirements of hyperspectral image fusion, the fusion process needs to preserve global features. Therefore, the selected loss function is...

[0109] (6)

[0110] in, For coefficients, The pixel similarity loss function is

[0111] ,

[0112] and

[0113] Let S be the structural similarity loss function. SSIM is a coefficient representing the structural similarity between two images. For the merged image, This is the original image.

[0114] (3) Input the coarsely registered hyperspectral image, extract image features at different scales through an autoencoder, obtain the fusion feature map of the m-th layer, and reconstruct the fused hyperspectral image through self-decoding; train the Spatio-VIT Block to obtain a high-precision hyperspectral fusion image.

[0115] Through the embodiments of the present invention, the technical solution of the present invention can improve the registration and fusion accuracy of satellite hyperspectral remote sensing images, and fuse spectral information into a single high-quality image.

[0116] Figure 4 This is a schematic diagram of a hyperspectral image processing device according to an embodiment of the present invention. The device includes a first module 401, a second module 402, a third module 403, and a fourth module 404.

[0117] The system comprises the following modules: a first module acquires a first hyperspectral image based on a hyperspectral image processing request and extracts its projection invariants, which are used to characterize the registration points and registration lines of the first hyperspectral image; a second module constructs a homography matrix based on the projection invariants; a third module constructs a rectangular network on the first hyperspectral image, constructs an energy function based on the homography matrix and projection invariants, and matches the first hyperspectral image using the energy function to obtain a second hyperspectral image, which is used to characterize a coarse registration image; and a fourth module performs fusion processing on the second hyperspectral image using a VIT deep learning fusion network to obtain a third hyperspectral image, which is used to characterize a high-precision image.

[0118] Exemplarily, with the cooperation of the first, second, third, and fourth modules in the device, the embodiment device can implement any of the aforementioned hyperspectral image processing methods, namely, acquiring a first hyperspectral image according to a hyperspectral image processing request; extracting the projection invariants of the first hyperspectral image, which are used to characterize the registration points and registration lines of the first hyperspectral image; constructing a homography matrix based on the projection invariants; constructing a rectangular network for the first hyperspectral image; constructing an energy function based on the homography matrix and projection invariants; matching the first hyperspectral image using the energy function to obtain a second hyperspectral image, which is used to characterize a coarsely registered image; and performing fusion processing on the second hyperspectral image using a VIT deep learning fusion network to obtain a third hyperspectral image, which is used to characterize a high-precision image. This invention improves image registration and fusion accuracy through point-line matching and deep learning methods, effectively fusing aerospace hyperspectral images and improving the target saliency of aerospace images.

[0119] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0120] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned hyperspectral image processing method.

[0121] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0122] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0124] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0125] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0128] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A hyperspectral image processing method, characterized in that, include: According to the hyperspectral image processing request, a first hyperspectral image is acquired, and the projection invariants of the first hyperspectral image are extracted. The projection invariants are used to characterize the registration points and registration lines of the first hyperspectral image. Construct a homography matrix based on the projection invariants; A rectangular network is constructed for the first hyperspectral image, and an energy function is constructed based on the homography matrix and the projection invariant. The first hyperspectral image is then matched using the energy function to obtain a second hyperspectral image, which is used to characterize the coarse registration image. The second hyperspectral image is fused using a VIT deep learning fusion network to obtain a third hyperspectral image, which is used to characterize a high-precision image. The step of extracting the projection invariants of the first hyperspectral image includes: using the SURF algorithm to search for the registration points from the first hyperspectral image; using the LSD algorithm to obtain registration line pairs from the first hyperspectral image; performing sub-region division based on the registration line pairs; wherein the sub-region division includes the original line segments obtained using the LSD algorithm; dividing the neighborhood of the original line segments into left and right neighborhoods based on the gradient direction; and when any pixel's distance from the line and its distance from the perpendicular bisector within the neighborhood of each line segment's line satisfy a preset value, the line is used as the registration line. The construction of the homography matrix based on the projection invariants includes: The RANSAC method is used to refine the registration points, and a homography matrix vector variable is constructed. While keeping the homography matrix vector variable to a minimum, the homography matrix of the transformation between the images to be matched is calculated, and the homography matrix is ​​calculated and its minimum value is obtained by using the SVD algorithm. The images to be matched are the first hyperspectral images corresponding to the registration points and the registration lines. The process of fusing the second hyperspectral image using a VIT deep learning fusion network to obtain a third hyperspectral image includes: A VIT-based deep learning fusion network is created with the second hyperspectral image as input; image features at different scales of the second hyperspectral image are extracted through the VIT deep learning fusion network, and a fusion feature map is obtained; the fusion feature map is reconstructed by self-decoding; the fusion hyperspectral image is processed by VIT-based deep learning to obtain the third hyperspectral image.

2. The hyperspectral image processing method according to claim 1, characterized in that, The method further includes: The number of features of the projection invariants is set, point-line invariants are constructed, and coplanar subregions are matched using point-line invariants to find the registration points and registration lines, thereby obtaining multiple projection invariants.

3. The hyperspectral image processing method according to claim 1, characterized in that, The step of constructing a rectangular network for the first hyperspectral image, constructing an energy function based on the homography matrix and the projection invariants, and matching the first hyperspectral image using the energy function to obtain a second hyperspectral image includes: A rectangular network is constructed for the image to be stitched, the rectangular network having grid vertex coordinate indices and original image grid vertex coordinates; Using the homography matrix, an energy function is constructed. The vertex coordinates of the distorted image grid are calculated while maintaining the lowest energy level in the energy function. A grid transformation is then used to obtain the distorted corrected image to be stitched. The stitching formula is as follows: in, The coordinates of the original image grid vertices. These are the coordinates of the distorted mesh vertices. To construct the energy function, This is a protection item for local and global lines. Used to control the overlap of matching point pairs and line pairs after splicing. The second hyperspectral image is obtained by maintaining the slope of the grid lines and uniform adjacent grids to control distortion during the stitching process.

4. The hyperspectral image processing method according to claim 1, characterized in that, The VIT deep learning fusion network also includes: When obtaining the fused feature map, global features are preserved through a loss function, which is: ， in, For coefficients, pixels The similarity loss function is ， in, For coefficients, Structural similarity loss function ， SSIM is used to represent the structural similarity between two images, where For the merged image, This is the original image.

5. A hyperspectral image processing device, characterized in that, include: The first module is used to acquire a first hyperspectral image according to a hyperspectral image processing request, and extract the projection invariants of the first hyperspectral image, wherein the projection invariants are used to characterize the registration points and registration lines of the first hyperspectral image. The second module is used to construct a homography matrix based on the projection invariants; The third module is used to construct a rectangular network for the first hyperspectral image, construct an energy function based on the homography matrix and the projection invariant, and match the first hyperspectral image through the energy function to obtain a second hyperspectral image, which is used to characterize the coarse registration image. The fourth module is used to perform fusion processing on the second hyperspectral image using a VIT deep learning fusion network to obtain a third hyperspectral image, which is used to characterize a high-precision image. The first module is further configured to use the SURF algorithm to search for the registration points from the first hyperspectral image; use the LSD algorithm to obtain registration line pairs from the first hyperspectral image; and perform sub-region division based on the registration line pairs. The sub-region division includes the original line segments obtained by the LSD algorithm, dividing the neighborhood of the original line segments into left and right neighborhoods according to the gradient direction. When any pixel's distance from the line and its distance from the vertical bisector in the neighborhood of the line of each line segment meet a preset value, the line is used as the registration line. The second module is further used to refine the registration points using the RANSAC method, construct homography matrix vector variables, calculate the homography matrix of the transformation between the images to be matched while keeping the homography matrix vector variables to a minimum, and calculate the homography matrix and obtain the minimum value by using the SVD algorithm, wherein the images to be matched are the first hyperspectral images corresponding to the registration points and the registration lines; The fourth module is also used to create a VIT-based deep learning fusion network with the second hyperspectral image as input; extract image features of different scales from the second hyperspectral image through the VIT deep learning fusion network and obtain a fusion feature map; reconstruct the fusion hyperspectral image by self-decoding the fusion feature map; and process the fusion hyperspectral image using VIT-based deep learning to obtain the third hyperspectral image.

6. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the hyperspectral image processing method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the hyperspectral image processing method as described in any one of claims 1-4.

Images