Hyperspectral imaging method and device, electronic equipment and storage medium

Abstract

The invention relates to a hyperspectral imaging method and device, electronic equipment and a storage medium. The hyperspectral imaging method comprises the steps of determining a rotary diffraction blurred image and an RGB image corresponding to a target object; determining an initial hyperspectral image corresponding to the target object according to the rotation diffraction blurred image; and a target hyperspectral image corresponding to the target object is determined according to the initial hyperspectral image and the RGB image by using a hyperspectral image reconstruction model, and the resolution corresponding to the target hyperspectral image is higher than the resolution corresponding to the initial hyperspectral image. According to the embodiment of the invention, the physical information of the rotation diffraction imaging process can be fully utilized, the rotation diffraction image is preliminarily reconstructed, the hyperspectral image reconstruction model and the RGB image are combined to carry out secondary reconstruction, the target hyperspectral image with high resolution is obtained, and the resolution of the target hyperspectral image is improved for different application scenes. The neural network model does not need to be retrained, and the application range and the mobility of the hyperspectral imaging method provided by the invention can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of spectral imaging technology, and in particular to a hyperspectral imaging method, device, electronic device, and storage medium. Background Art

[0002] A common spectral imaging method in the prior art is to encode spectral information to obtain an encoded image, and then reconstruct the encoded image to obtain a hyperspectral image. However, the commonly used image reconstruction method in the prior art relies on a neural network model and cannot utilize the physical information of the encoding imaging process. Therefore, for different application scenarios, the neural network model needs to be retrained, which lacks portability. Summary of the invention

[0003] The present invention provides a technical solution for a hyperspectral imaging method, device, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, a hyperspectral imaging method is provided, comprising: determining a rotational diffraction blurred image and an RGB image corresponding to a target object; determining an initial hyperspectral image corresponding to the target object according to the rotational diffraction blurred image; and determining a target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image using a hyperspectral image reconstruction model, wherein a resolution corresponding to the target hyperspectral image is higher than a resolution corresponding to the initial hyperspectral image.

[0005] In a possible implementation, determining the initial hyperspectral image corresponding to the target object based on the rotational diffraction blurred image includes: based on a two-step iterative shrinkage threshold algorithm, utilizing physical information of a rotational diffraction imaging process, iteratively reconstructing the rotational diffraction blurred image to determine the initial hyperspectral image.

[0006] In a possible implementation, the using a hyperspectral image reconstruction model to determine a target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image includes: aligning the initial hyperspectral image and the RGB image to determine an aligned image group; and inputting the aligned image group into the hyperspectral image reconstruction model to determine the target hyperspectral image.

[0007] In a possible implementation manner, aligning the initial hyperspectral image and the RGB image to determine the aligned image group includes: performing feature point matching on the initial hyperspectral image and the RGB image to determine a feature point matching result; and aligning the initial hyperspectral image and the RGB image according to the feature point matching result to determine the aligned image group.

[0008] In a possible implementation, the rotational diffraction blurred image is obtained using an imaging system based on the principle of rotational diffraction; the method also includes: calibrating the imaging system based on the principle of rotational diffraction, determining the point spread function corresponding to each wavelength channel of the imaging system based on the principle of rotational diffraction within a target wavelength range, wherein the point spread function is used to determine a rotational diffraction imaging model, and the rotational diffraction imaging model is used to determine physical information of the rotational diffraction imaging process.

[0009] In one possible implementation, the imaging system based on the principle of rotational diffraction includes: a beam splitter, a rotating diffraction element, a first image sensor, an optical lens, and a second image sensor, wherein the rotating diffraction element is obtained by processing a target material according to target lens surface parameters, wherein the target lens surface parameters are determined based on a preset number of periods, a target wavelength range, and a material refractive index corresponding to the target material.

[0010] In a possible implementation manner, the focal length corresponding to the rotating diffraction element is the same as the focal length corresponding to the optical lens, and the resolution corresponding to the first image sensor is the same as the resolution corresponding to the second image sensor.

[0011] According to another aspect of the present disclosure, a hyperspectral imaging device is provided, comprising: an image acquisition module, used to determine a rotational diffraction blurred image and an RGB image corresponding to a target object; a physical reconstruction module, used to determine an initial hyperspectral image corresponding to the target object according to the rotational diffraction blurred image; and a high-resolution reconstruction module, used to determine a target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image using a hyperspectral image reconstruction model, wherein the resolution corresponding to the target hyperspectral image is higher than the resolution corresponding to the initial hyperspectral image.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0013] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0014] In the disclosed embodiment, after determining the rotational diffraction blurred image and the RGB image corresponding to the target object, the initial hyperspectral image corresponding to the target object can be determined according to the rotational diffraction blurred image, thereby making full use of the physical information of the rotational diffraction imaging process; utilizing the hyperspectral image reconstruction model, the resolution of the initial hyperspectral image is improved based on the image information corresponding to the RGB image, and a target hyperspectral image having a higher resolution than the initial hyperspectral image is obtained, thereby reducing the requirements for spatial resolution of the reconstruction of the rotational diffraction blurred image and improving the efficiency of hyperspectral imaging; and, in the case where the imaging system does not change, there is no need to retrain the hyperspectral image reconstruction model for different application scenarios, thereby improving the scope of application and portability of hyperspectral imaging.

[0015] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0017] Figure 1 A flow chart of a hyperspectral imaging method according to an embodiment of the present disclosure is shown;

[0018] Figure 2 A schematic structural diagram of an imaging system based on the principle of rotational diffraction according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram showing the relationship between the wavelength of light and the polar angle of a rotating diffraction element when the preset period number is 2 according to an embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram showing a surface shape corresponding to a rotating diffraction element when a preset period number is 2 according to an embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram showing a process of hyperspectral image reconstruction according to an embodiment of the present disclosure is shown;

[0022] Figure 6 A block diagram of a hyperspectral imaging device according to an embodiment of the present disclosure is shown;

[0023] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0027] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0028] Traditional hyperspectral imaging systems include systems based on narrowband filters, systems based on Michael interferometers, systems based on dispersive elements, etc. According to the Nyquist sampling theory, these imaging systems cannot take into account spectral resolution, spatial resolution, and temporal resolution at the same time. Therefore, in order to obtain all the two-dimensional information and spectral information of the target object, it is necessary to scan the target object in the spatial dimension or spectral dimension, which increases the time of hyperspectral imaging. In order to overcome the mutual constraints between spectral resolution, spatial resolution, and temporal resolution, computational spectral imaging technology is proposed in the prior art, that is, encoding the input spectral information and reconstructing the spectral information into a hyperspectral image through an algorithm to achieve high-resolution real-time hyperspectral imaging.

[0029] Common spectral encoding methods in the prior art include: diffraction element encoding, aperture encoding, filter encoding, etc. The rotating diffraction encoding device based on the diffraction optical element is a new type of diffraction element. After the light with different wavelengths passes through the rotating diffraction encoding device, the corresponding point spread function (PSF) can present a similar shape with rotation characteristics. Through this diffraction element, the volume of the diffraction element encoding imaging system can be reduced, the imaging system can be miniaturized, and the snapshot imaging function can be realized.

[0030] Common spectral information reconstruction algorithms in the prior art include reconstruction algorithms based on imaging models, reconstruction algorithms based on neural networks, etc. Among them, reconstruction algorithms based on imaging models can make full use of the information of the imaging process and can adapt to a variety of different application scenarios, but the reconstruction efficiency is low; reconstruction algorithms based on neural networks have higher reconstruction efficiency, but for different application scenarios, the neural network model usually needs to be retrained, the training process of the neural network model takes a long time, and the flexibility of processing low-level visual tasks is poor.

[0031] In the prior art, methods for improving the performance of neural network-based reconstruction algorithms mainly focus on increasing the amount of sample data used to train neural network models. Among them, a common way is to add RGB images as a guide, using the high resolution of RGB images to make up for the shortcomings of hyperspectral images in spatial resolution and improve image quality. However, these methods usually rely entirely on neural network models, ignoring the information of the imaging process, and still cannot avoid the problem of needing to retrain neural network models for different application scenarios, and have weak transferability. In addition, when fusing RGB images, it is necessary to accurately align RGB images and hyperspectral images, and the calibration and position adjustment of the optical components of the imaging system are required to be high.

[0032] In view of this, the present disclosure provides a hyperspectral imaging method, which can perform preliminary reconstruction of the rotating diffraction image based on the physical model of rotating diffraction element imaging, thereby making full use of the information of the rotating diffraction imaging process, and then combining the neural network model and the RGB image for secondary reconstruction to obtain a target hyperspectral image with a higher resolution. In the case where the imaging system does not change, for different application scenarios, there is no need to retrain the neural network model, which can improve the scope of application and portability of the hyperspectral imaging method provided by the present disclosure. The hyperspectral imaging method provided by the present disclosure is described in detail below.

[0033] Figure 1 A flow chart of a hyperspectral imaging method according to an embodiment of the present disclosure is shown. The hyperspectral imaging method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The hyperspectral imaging method can be implemented by a processor calling computer-readable instructions stored in a memory. Alternatively, the hyperspectral imaging method can be executed by a server. Figure 1 As shown, the hyperspectral imaging method includes:

[0034] In step S11, the rotational diffraction blurred image and the RGB image corresponding to the target object are determined.

[0035] The target object here may represent any object that needs to be imaged by hyperspectral imaging. The specific form and quantity of the target object may be flexibly set according to actual usage requirements, and the present disclosure does not make any specific limitation on this.

[0036] The rotational diffraction blurred image corresponding to the target object can represent an image obtained by encoding the spectral information corresponding to the target object based on the principle of rotational diffraction. Its specific form can refer to the implementation method in the relevant technology, and the present disclosure does not make any specific limitation on it.

[0037] The specific method of determining the rotational diffraction blurred image and the RGB image corresponding to the target object can refer to the implementation methods in the relevant technology, and the present disclosure does not make any specific limitations on this.

[0038] In one possible implementation, a rotational diffraction blurred image is obtained using an imaging system based on the principle of rotational diffraction; the method also includes: calibrating the imaging system based on the principle of rotational diffraction, determining the point spread function corresponding to each wavelength channel of the imaging system based on the principle of rotational diffraction within the target wavelength range, wherein the point spread function is used to determine a rotational diffraction imaging model, and the rotational diffraction imaging model is used to determine physical information of the rotational diffraction imaging process.

[0039] The specific form of the imaging system based on the principle of rotational diffraction here can refer to the implementation methods in the relevant technology, and the present disclosure does not make any specific limitations on this.

[0040] In one possible implementation, an imaging system based on the principle of rotational diffraction includes: a beam splitter, a rotating diffraction element, a first image sensor, an optical lens, and a second image sensor. The rotating diffraction element is obtained by processing a target material according to target lens surface parameters, wherein the target lens surface parameters are determined based on a preset number of periods, a target wavelength range, and a material refractive index corresponding to the target material.

[0041] Figure 2 FIG. 1 is a schematic diagram showing a structure of an imaging system based on the principle of rotational diffraction according to an embodiment of the present disclosure. Figure 2As shown, the imaging system based on the rotation diffraction principle includes a beam splitter 201, a rotation diffraction element 202, a first image sensor 203, an optical lens 204, and a second image sensor 205; the rotation diffraction element 202 is arranged on the first image sensor 203, and is collinear with the transmission optical axis corresponding to the beam splitter 201, so as to collect the rotation diffraction blurred image corresponding to the target object; the optical lens 204 is arranged on the second image sensor 204, and is collinear with the reflection optical axis corresponding to the beam splitter 201, so as to collect the RGB image corresponding to the target object. The specific forms of the beam splitter 201, the first image sensor 203, the optical lens 204, and the second image sensor 205 can refer to the implementation in the related art, and the present disclosure does not specifically limit this.

[0042] The rotating diffractive element 202 is obtained by processing the target material according to the target lens surface shape parameters. The target material here can refer to the material used to process the target diffractive lens. Its specific form can refer to the implementation method in the relevant technology, and the present disclosure does not make specific restrictions on this. The target lens surface shape parameters can reflect the shape characteristics and size characteristics corresponding to the rotating diffractive element 202. Its specific content can be flexibly set according to actual usage requirements. For example, it can include the diameter corresponding to the rotating diffractive element 202, the thickness corresponding to any position on the rotating diffractive element 202, etc. The present disclosure does not make specific restrictions on this.

[0043] The target lens surface parameters can be determined according to the preset period number, the target wavelength range and the material refractive index corresponding to the target material. Among them, the preset period number can represent the repetition period corresponding to the rotating diffraction element 202, and can be used to divide the 360° of the rotating diffraction element 202 into a plurality of identical repetition areas to enrich the point spread function of the light after passing through the rotating diffraction element 202. The specific value can be flexibly set according to the actual use requirements. For example, the preset period number can be set to be equal to 2, etc., and the present disclosure does not make specific restrictions on this. The target wavelength range can represent the working range of the rotating diffraction element, and the specific value can be flexibly set according to the actual use requirements. The present disclosure does not make specific restrictions on this.

[0044] Specifically, the target lens surface parameters may include: the thickness corresponding to any position on the rotating diffraction element 202. Different thicknesses may be set for different angular positions on the rotating diffraction element 202, so that after light with different wavelengths passes through the rotating diffraction element 202, the angles corresponding to the diffraction main maximum positions on the imaging plane are different, thereby making the point spread functions corresponding to the light with different wavelengths present similar shapes, but with a certain rotation angle between them.

[0045] Figure 3A schematic diagram showing the relationship between the wavelength of light and the polar angle of the rotating diffraction element when the preset period number is 2 according to an embodiment of the present disclosure. Figure 3 As shown, when the preset number of periods is 2, in the range of the polar angle corresponding to the rotating diffraction element being greater than or equal to 0° and less than 180°, the wavelength changes linearly with the polar angle; when the polar angle corresponding to the rotating diffraction element is 180°, the wavelength changes suddenly; in the range of the polar angle corresponding to the rotating diffraction element being greater than 180° and less than 360°, the wavelength changes linearly with the polar angle.

[0046] Then the relationship between the wavelength of the light and the angular position corresponding to the rotating diffraction element 202 can be expressed as formula (1):

[0047]

[0048] Where θ represents the polar angle; λ(θ) represents the design wavelength corresponding to the angular position θ; λ max Indicates the maximum design wavelength corresponding to the target wavelength range; λ min Indicates the minimum design wavelength corresponding to the target wavelength range; N indicates the preset number of cycles.

[0049] The total optical path difference between the two light beams emitted from different positions on the rotating diffraction element 202 can be expressed as the sum of the first optical path difference generated by the rotating diffraction element 202 modulating the light field and the second optical path difference generated by the different propagation paths of the two light beams in the air. Assuming that the first light beam is emitted from the geometric center of the rotating diffraction element 202 and the second light beam is emitted from a position r away from the geometric center of the rotating diffraction element 202, the total optical path difference between the first light beam and the second light beam can be expressed as formula (2):

[0050]

[0051] Where, L represents the total optical path difference; L h Represents the first optical path difference; L g represents the second optical path difference; f represents the focusing distance corresponding to the rotating diffraction element; n λ represents the refractive index of the target material to the light with wavelength λ; n0 represents the refractive index of air to the light with wavelength λ; h r represents the thickness corresponding to the position r on the rotating diffraction element from the geometric center; h0 represents the thickness corresponding to the geometric center of the rotating diffraction element.

[0052] The condition for diffraction between the first beam and the second beam can be expressed as formula (3):

[0053]

[0054] Here, n can be any natural number.

[0055] According to the above formulas (1) to (3), the thickness corresponding to any position on the rotating diffractive element 202 can be determined. Specifically, the thickness corresponding to any position on the rotating diffractive element 202 can be expressed as formula (4):

[0056]

[0057] Among them, h(r,θ) represents the thickness corresponding to the position on the rotating diffraction element with a polar angle of θ and a distance r from the geometric center of the rotating diffraction element; Δh(r,θ) represents the relative thickness between the position on the rotating diffraction element with a polar angle of θ and a distance r from the geometric center of the rotating diffraction element and the geometric center of the rotating diffraction element; h(0,0) represents the thickness corresponding to the geometric center of the rotating diffraction element.

[0058] Figure 4 A schematic diagram showing a surface shape corresponding to a rotating diffraction element when the preset period number is 2 according to an embodiment of the present disclosure. Figure 4 As shown, when the preset period number is 2, the rotating diffraction element is divided into two repeated areas, and the thicknesses at different positions relative to the geometric center of the rotating diffraction element are different.

[0059] In a possible implementation manner, the focal length corresponding to the rotating diffraction element is the same as the focal length corresponding to the optical lens, and the resolution corresponding to the first image sensor is the same as the resolution corresponding to the second image sensor.

[0060] In order to simplify the complexity of the image processing process, the focal length corresponding to the rotating diffraction element can be set to be the same as the focal length corresponding to the optical lens, and the resolution corresponding to the first image sensor can be set to be the same as the resolution corresponding to the second image sensor, so that the image size and resolution corresponding to the rotational diffraction blurred image are the same as the image size and resolution corresponding to the RGB image, avoiding the need to perform additional normalization operations on the rotational diffraction blurred image and the RGB image during subsequent image processing.

[0061] Among them, the specific values ​​of the focal length corresponding to the rotating diffraction element and the focal length corresponding to the optical lens can be flexibly set according to actual usage requirements, and the present disclosure does not make specific limitations on this; the specific values ​​of the resolution corresponding to the first image sensor and the resolution corresponding to the second image sensor can be flexibly set according to actual usage requirements, and the present disclosure does not make specific limitations on this.

[0062] After determining the parameters of each optical element in the imaging system based on the rotational diffraction principle, the imaging system based on the rotational diffraction principle can be calibrated to determine the point spread function corresponding to each wavelength channel in the target wavelength range of the imaging system based on the rotational diffraction principle, wherein the point spread function is used to determine the rotational diffraction imaging model, and the rotational diffraction imaging model is used to determine the physical information of the rotational diffraction imaging process.

[0063] The specific method of calibrating the imaging system can refer to the implementation methods in the relevant technology, and the present disclosure does not make any specific limitations on this.

[0064] The rotational diffraction blurred image can be expressed as the image corresponding to different wavelength channels, which is the result of convolution and superposition of the point spread function corresponding to each wavelength channel. Based on this, the rotational diffraction imaging model corresponding to the rotational diffraction blurred image can be expressed as formula (5):

[0065]

[0066] Among them, J c (x, y) represents the rotational diffraction blurred image; Ω c (λ) represents the sensitivity of the first image sensor to light with a wavelength of λ; the subscript c represents the color channel, c∈{R,G,B}, where R represents the red channel, G represents the green channel, and B represents the blue channel; I λ represents the incident light field image corresponding to the light with wavelength λ; * represents the convolution operation; p λ represents the point spread function corresponding to the wavelength channel λ; n(x,y) represents the noise; (x,y) represents the position with coordinates (x,y) on the imaging plane.

[0067] RGB images can also be expressed as the superposition of images corresponding to different wavelength channels, and the corresponding imaging model can be expressed as formula (6):

[0068]

[0069] Among them, J′ c (x, y) represents an RGB image.

[0070] Through the above-mentioned imaging system based on the principle of rotational diffraction, the synchronous acquisition of the rotational diffraction blurred image and the RGB image can be realized to ensure the consistency of the rotational diffraction blurred image and the RGB image. Further, in order to reduce the difficulty of the subsequent image reconstruction process, when using the imaging system based on the principle of rotational diffraction for image acquisition, the field of view corresponding to the first image sensor and the field of view corresponding to the second image sensor can be set to be the same.

[0071] In step S12, an initial hyperspectral image corresponding to the target object is determined according to the rotational diffraction blurred image.

[0072] Based on the rotational diffraction imaging model, the information of the rotational diffraction imaging process can be used to preliminarily reconstruct the rotational diffraction blurred image separately to determine an initial hyperspectral image with a lower resolution. The specific method of preliminarily reconstructing the rotational diffraction blurred image can refer to the implementation in the related art, and the present disclosure does not specifically limit this.

[0073] The process of determining the initial hyperspectral image corresponding to the target object according to the rotational diffraction blurred image will be described in detail later in conjunction with possible implementation methods of the present disclosure, and will not be elaborated here.

[0074] In step S13, a target hyperspectral image corresponding to the target object is determined according to the initial hyperspectral image and the RGB image using a hyperspectral image reconstruction model, wherein the resolution corresponding to the target hyperspectral image is higher than the resolution corresponding to the initial hyperspectral image.

[0075] After the initial hyperspectral image is determined, the initial hyperspectral image can be reconstructed using the image information of the RGB image based on the hyperspectral image reconstruction model to obtain a target hyperspectral image with a higher resolution than the initial hyperspectral image. The specific form of the hyperspectral image reconstruction model can refer to the implementation in the related art, and the present disclosure does not specifically limit this.

[0076] In one possible implementation, multiple hyperspectral image reconstruction models corresponding to different neural network frameworks may be pre-selected, and for any hyperspectral image reconstruction model corresponding to the neural network framework, the reconstruction performance of the hyperspectral image reconstruction model corresponding to the neural network framework may be evaluated based on a sample image set. By comparing the performances of hyperspectral image reconstruction models corresponding to different neural network frameworks, the hyperspectral image reconstruction model with the best performance may be determined for reconstructing the initial hyperspectral image and the RGB image.

[0077] The following text will describe in detail the process of using the hyperspectral image reconstruction model to determine the target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image in combination with possible implementation methods of the present disclosure, which will not be described in detail here.

[0078] In the disclosed embodiment, after determining the rotational diffraction blurred image and the RGB image corresponding to the target object, the initial hyperspectral image corresponding to the target object can be determined according to the rotational diffraction blurred image, thereby making full use of the physical information of the rotational diffraction imaging process; utilizing the hyperspectral image reconstruction model, the resolution of the initial hyperspectral image is improved based on the image information corresponding to the RGB image, and a target hyperspectral image having a higher resolution than the initial hyperspectral image is obtained, thereby reducing the requirements for spatial resolution of the reconstruction of the rotational diffraction blurred image and improving the efficiency of hyperspectral imaging; and, in the case where the imaging system does not change, there is no need to retrain the hyperspectral image reconstruction model for different application scenarios, thereby improving the scope of application and portability of hyperspectral imaging.

[0079] In a possible implementation, an initial hyperspectral image corresponding to the target object is determined based on the rotational diffraction blurred image, including: based on a two-step iterative shrinkage threshold algorithm, using physical information of the rotational diffraction imaging process, iteratively reconstructing the rotational diffraction blurred image to determine the initial hyperspectral image.

[0080] Specifically, before the rotational diffraction blurred image is preliminarily reconstructed, the rotational diffraction blurred image may be compressed to further reduce the resolution corresponding to the rotational diffraction blurred image, so as to increase the speed of subsequent preliminary reconstruction.

[0081] The two-step iterative shrinkage threshold algorithm (Two-Step Iterative Shrinkage, TwIST) can make full use of the physical information of the rotational diffraction imaging process, perform multiple iterations on the rotational diffraction blurred image, and obtain the initial hyperspectral image. The iterative process of the TwIST algorithm can be expressed as formula (7):

[0082] x t+1 =(1-α)x t-1 +(α-β)x t +βp L (x t ) (7)

[0083] Among them, x t+1 represents the result of the t+1th iteration of the rotational diffraction blurred image; x t-1 represents the result of the t-1th iteration of the rotational diffraction blurred image; x t represents the result of the t-th iteration of the rotational diffraction blurred image; p L is a proximal gradient descent operator, α, β and L represent iteration parameters; represents a function related to the parameter L and the regularization term R(x), and the specific form of the regularization term R(x) can be flexibly set according to actual usage requirements. For example, it can be a total variation regularization term, etc., and the present disclosure does not make specific limitations on this.

[0084] In a possible implementation, a hyperspectral image reconstruction model is used to determine a target hyperspectral image corresponding to a target object according to an initial hyperspectral image and an RGB image, including: aligning the initial hyperspectral image and the RGB image to determine an aligned image group; and inputting the aligned image group into the hyperspectral image reconstruction model to determine the target hyperspectral image.

[0085] In order to improve the difficulty of image reconstruction of the initial hyperspectral image and the RGB image and improve the accuracy of image reconstruction, the initial hyperspectral image and the RGB image can be aligned to determine the aligned image group. The specific method of image alignment can refer to the implementation method in the related art. For example, the ORB detector in OpenCV can be used to directly align the initial hyperspectral image and the RGB image, and the present disclosure does not specifically limit this.

[0086] In a possible implementation, the initial hyperspectral image and the RGB image are aligned to determine an aligned image group, including: performing feature point matching on the initial hyperspectral image and the RGB image to determine a feature point matching result; and aligning the initial hyperspectral image and the RGB image according to the feature point matching result to determine an aligned image group.

[0087] The key point of image alignment between any two images is to determine the homography matrix (Homography) of the image conversion between the two images, and the determination of the parameters of the homography matrix depends on the correspondence between the same physical points between the two images. Therefore, feature point extraction can be performed on the initial hyperspectral image and the RGB image respectively, and then feature point matching can be performed to determine the feature point matching results of the initial hyperspectral image and the RGB image. Among them, the specific method of extracting and matching feature points on the initial hyperspectral image and the RGB image can refer to the implementation method in the relevant technology, and the present disclosure does not make specific limitations on this.

[0088] According to the feature point matching results, the homography matrix between the initial hyperspectral image and the RGB image can be calculated, and then the initial hyperspectral image and the RGB image can be aligned to determine the aligned image group.

[0089] By achieving image alignment through the above process, the accuracy and reliability of subsequent image reconstruction can be ensured while reducing the requirements on the processing accuracy of each optical element in the imaging system based on the principle of rotational diffraction and the requirements on the calibration accuracy of the imaging system.

[0090] By inputting the aligned image group into the hyperspectral image reconstruction model, a complete hyperspectral image reconstruction can be achieved to obtain a target hyperspectral image with higher resolution.

[0091] Figure 5 FIG. 2 is a schematic diagram showing a process of hyperspectral image reconstruction according to an embodiment of the present disclosure. Figure 5 As shown, based on the two-step iterative shrinkage threshold algorithm, the rotational diffraction blurred image is preliminarily reconstructed to obtain an initial hyperspectral image with a lower resolution; the RGB image with extremely high resolution and the initial hyperspectral image are aligned to determine the aligned image group, which is input into the hyperspectral image reconstruction model to obtain a target hyperspectral image with a higher resolution.

[0092] In the embodiment of the present disclosure, after determining the rotational diffraction blurred image and RGB image corresponding to the target object, the initial hyperspectral image corresponding to the target object can be determined based on the rotational diffraction blurred image, making full use of the physical information of the rotational diffraction imaging process; using the hyperspectral image reconstruction model, the resolution of the initial hyperspectral image is improved based on the image information corresponding to the RGB image, and a target hyperspectral image with a higher resolution than the initial hyperspectral image is obtained, thereby reducing the requirements for spatial resolution of the reconstruction of the rotational diffraction blurred image and improving the efficiency of hyperspectral imaging; and, in the case where the imaging system does not change, there is no need to retrain the hyperspectral image reconstruction model for different application scenarios, thereby improving the scope of application and portability of hyperspectral imaging. In addition, in the embodiment of the present disclosure, the initial hyperspectral image and the RGB image are aligned by an algorithm, which can reduce the requirements for the processing accuracy of each optical element in the imaging system based on the rotational diffraction principle, as well as the requirements for the calibration accuracy of the imaging system.

[0093] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0094] In addition, the present disclosure also provides a hyperspectral imaging device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any hyperspectral imaging method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0095] Figure 6 FIG. 5 is a block diagram of a hyperspectral imaging device according to an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes:

[0096] An image acquisition module 601 is used to determine a rotational diffraction blurred image and an RGB image corresponding to a target object;

[0097] A physical reconstruction module 602 is used to determine an initial hyperspectral image corresponding to the target object according to the rotation diffraction blurred image;

[0098] The high-resolution reconstruction module 603 is used to determine a target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image using the hyperspectral image reconstruction model, wherein the resolution corresponding to the target hyperspectral image is higher than the resolution corresponding to the initial hyperspectral image.

[0099] In a possible implementation, the physical reconstruction module 602 is specifically configured to iteratively reconstruct the rotational diffraction blurred image based on a two-step iterative shrinkage threshold algorithm to determine an initial hyperspectral image.

[0100] In a possible implementation, the high-resolution reconstruction module 603 is specifically used to: align the initial hyperspectral image and the RGB image to determine an aligned image group; and input the aligned image group into a hyperspectral image reconstruction model to determine a target hyperspectral image.

[0101] In a possible implementation, the high-resolution reconstruction module 603 is further used to: perform feature point matching on the initial hyperspectral image and the RGB image to determine a feature point matching result; and align the initial hyperspectral image and the RGB image according to the feature point matching result to determine an aligned image group.

[0102] In one possible implementation, the rotational diffraction blurred image is obtained using an imaging system based on the principle of rotational diffraction; the device 600 also includes an imaging system calibration module, which is used to: calibrate the imaging system based on the principle of rotational diffraction, and determine the point spread function corresponding to each wavelength channel of the imaging system based on the principle of rotational diffraction within the target wavelength range, wherein the point spread function is used to determine the rotational diffraction imaging model, and the rotational diffraction imaging model is used to determine the physical information of the rotational diffraction imaging process.

[0103] In one possible implementation, an imaging system based on the principle of rotational diffraction includes: a beam splitter, a rotating diffraction element, a first image sensor, an optical lens, and a second image sensor. The rotating diffraction element is obtained by processing a target material according to target lens surface parameters, wherein the target lens surface parameters are determined based on a preset number of periods, a target wavelength range, and a material refractive index corresponding to the target material.

[0104] In a possible implementation manner, the focal length corresponding to the rotating diffraction element is the same as the focal length corresponding to the optical lens, and the resolution corresponding to the first image sensor is the same as the resolution corresponding to the second image sensor.

[0105] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0106] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0107] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0108] Figure 7 1900 is a block diagram of an electronic device according to an embodiment of the present disclosure. For example, the apparatus 1900 may be provided as a server or a terminal device. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0109] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0110] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0111] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0112] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0113] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0114] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0115] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0116] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0117] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0118] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0119] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A hyperspectral imaging method, characterized in that: include: Determine the rotational diffraction blurred image and RGB image corresponding to the target object; Determining an initial hyperspectral image corresponding to the target object according to the rotational diffraction blurred image; A target hyperspectral image corresponding to the target object is determined according to the initial hyperspectral image and the RGB image by using a hyperspectral image reconstruction model, wherein a resolution corresponding to the target hyperspectral image is higher than a resolution corresponding to the initial hyperspectral image.

2. The method according to claim 1, characterized in that Determining the initial hyperspectral image corresponding to the target object according to the rotational diffraction blurred image includes: Based on a two-step iterative shrinkage threshold algorithm and utilizing physical information of a rotational diffraction imaging process, the rotational diffraction blurred image is iteratively reconstructed to determine the initial hyperspectral image.

3. The method according to claim 1 or 2, characterized in that: The method of using a hyperspectral image reconstruction model to determine a target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image includes: Aligning the initial hyperspectral image and the RGB image to determine an aligned image group; The aligned image group is input into the hyperspectral image reconstruction model to determine the target hyperspectral image.

4. The method according to claim 3, characterized in that The aligning the initial hyperspectral image and the RGB image to determine an aligned image group includes: Performing feature point matching on the initial hyperspectral image and the RGB image to determine a feature point matching result; According to the feature point matching result, the initial hyperspectral image and the RGB image are aligned to determine the aligned image group.

5. The method according to claim 1 or 2, characterized in that: The rotational diffraction blurred image is obtained by using an imaging system based on the rotational diffraction principle; The method further comprises: The imaging system based on the rotational diffraction principle is calibrated to determine the point spread function corresponding to each wavelength channel of the imaging system based on the rotational diffraction principle within the target wavelength range, wherein the point spread function is used to determine the rotational diffraction imaging model, and the rotational diffraction imaging model is used to determine the physical information of the rotational diffraction imaging process.

6. The method according to claim 5, characterized in that The imaging system based on the principle of rotational diffraction includes: a beam splitter, a rotational diffraction element, a first image sensor, an optical lens and a second image sensor. The rotational diffraction element is obtained by processing a target material according to target lens surface parameters, wherein the target lens surface parameters are determined according to a preset number of periods, a target wavelength range and a material refractive index corresponding to the target material.

7. The method according to claim 6, characterized in that The focal length corresponding to the rotating diffraction element is the same as the focal length corresponding to the optical lens, and the resolution corresponding to the first image sensor is the same as the resolution corresponding to the second image sensor.

8. A hyperspectral imaging device, characterized in that: include: An image acquisition module, used to determine a rotational diffraction blurred image and an RGB image corresponding to a target object; A physical reconstruction module, used for determining an initial hyperspectral image corresponding to the target object according to the rotation diffraction blurred image; The high-resolution reconstruction module is used to determine a target hyperspectral image corresponding to the target object according to the initial hyperspectral image and the RGB image by using a hyperspectral image reconstruction model, wherein the resolution corresponding to the target hyperspectral image is higher than the resolution corresponding to the initial hyperspectral image.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images