A snapshot-type hyperspectral three-dimensional skin imaging method and device

Abstract

This invention discloses a snapshot-type hyperspectral three-dimensional skin imaging method and device, belonging to the field of biomedical imaging technology. The method includes: constructing a snapshot-type spectral-light field coupled imaging system containing a light source and an acquisition model; achieving full-domain skin illumination with broadband illumination; synchronously coupling and acquiring spectral and three-dimensional spatial information through the synergistic action of a lens group and a hyperspectral sensor to generate spectral coupled light field data; establishing a correlation dataset between the spectral coupled light field image and the hyperspectral light field image; training a spectral reconstruction network and solving the data; reconstructing the hyperspectral light field image and obtaining multi-view images through macro-pixel sampling; using a light field super-resolution network to improve image resolution; selecting the optimal skin penetration band image for stereo matching to obtain a target band depth map; and fusing the high-resolution center-view images from each channel with the target band depth map to complete the generation of a hyperspectral three-dimensional image. This invention achieves single-exposure hyperspectral three-dimensional imaging of the superficial skin layer.

Description

Technical Field

[0001] This invention relates to the field of biomedical imaging technology, and in particular to a snapshot-type hyperspectral three-dimensional skin imaging method and apparatus. Background Technology

[0002] Dermatoscopes are commonly used optical imaging devices in the field of dermatological diagnosis, primarily for the examination and evaluation of skin lesions such as melanoma and basal cell carcinoma. Utilizing high magnification, these devices can reveal minute lesions on the skin surface that are difficult to detect with the naked eye, providing crucial reference information for clinical diagnosis. Traditional dermatoscopes rely on visible light illumination to acquire detailed images of the skin surface through a magnifying lens; however, in practical applications, they suffer from significant drawbacks, such as insufficient acquisition of depth and spectral information.

[0003] On the one hand, traditional dermoscopy can only output two-dimensional planar images, failing to capture the layered structural information within the skin. This makes it difficult to accurately assess the depth and extent of lesion infiltration, especially since malignant skin lesions often exhibit multi-level invasion characteristics, severely limiting the comprehensiveness of diagnosis due to the two-dimensional imaging mode. On the other hand, traditional dermoscopy has weak spectral information acquisition capabilities, typically relying on scanning imaging modes to obtain hyperspectral images, resulting in low imaging efficiency. Components in skin tissue, such as melanin, collagen, and hemoglobin, possess differentiated absorption and scattering characteristics in the visible and infrared bands. The vascular distribution, cell density, and metabolic level of malignant lesions differ significantly from normal tissues, exhibiting specific spectral responses. The lack of spectral information makes it difficult for traditional dermoscopy to effectively distinguish subtle differences between lesions and normal tissues, resulting in low sensitivity in the detection of early, occult skin lesions and hindering the improvement of diagnostic accuracy.

[0004] Therefore, there is an urgent need for an optical diagnostic device that can simultaneously acquire skin tissue depth information and hyperspectral information and has rapid imaging capabilities, in order to overcome the technical limitations of traditional dermoscopy and improve the accuracy and efficiency of skin lesion diagnosis. Summary of the Invention

[0005] The main objective of this invention is to provide a snapshot-type hyperspectral three-dimensional skin imaging method.

[0006] Another objective of this invention is to provide a snapshot-type hyperspectral three-dimensional skin imaging device.

[0007] The third objective of this invention is to provide an electronic device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention provides a snapshot-type hyperspectral three-dimensional skin imaging method, comprising:

[0010] S1. Construct a snapshot-type spectral-light field coupled imaging system containing a light source model and an acquisition model. The light source model outputs broadband illumination to achieve full-area skin illumination. The acquisition model works in conjunction with a hyperspectral sensor through a lens group to synchronously couple and acquire skin spectral and three-dimensional spatial information to generate spectral coupled light field data. S2, establish a correlation dataset between the spectral coupled light field image and the hyperspectral light field image, train a spectral reconstruction network based on the correlation dataset, solve the spectral coupled light field data and reconstruct the hyperspectral light field image through the spectral reconstruction network, and obtain multi-view hyperspectral images from different perspectives through macro-pixel multi-position sampling. S3 uses a light field super-resolution network to perform super-resolution processing on multi-view hyperspectral images to obtain high-resolution multi-view hyperspectral images corresponding to each spectral band, and selects the high-resolution multi-view image corresponding to the optimal penetration band for stereo matching to generate a skin superficial tissue depth map. S4. Select the target band image with the best skin penetration from the high-resolution multi-view images of each spectral channel, perform stereo matching based on the center view image to obtain the target band depth map, and fuse the high-resolution center view images of each channel with the target band depth map to complete the generation of the hyperspectral three-dimensional image.

[0011] Optionally, the lens group includes a main lens and a microlens array. The microlens array is composed of multiple sub-microlenses with identical structures arranged in a periodic array. The microlens array is fixedly set in the imaging optical path between the main lens and the hyperspectral sensor, providing multi-view light field information acquisition function for the acquisition model. There are two switchable optical path assembly structures between the microlens array and the hyperspectral sensor. One is to directly attach the microlens array to the photosensitive surface of the hyperspectral sensor, so that the photosensitive surface of the hyperspectral sensor is precisely located at the focal plane of the microlens array. The other is to connect a relay imaging system between the microlens array and the hyperspectral sensor, so that the focal plane of the microlens array and the photosensitive surface of the hyperspectral sensor form a conjugate surface, ensuring the distortion-free transmission of light field information. The number of sub-views acquired is determined by the sensor pixel array size corresponding to a single sub-microlens. When using a pixel array, the number of sub-viewpoints acquired is , where N is a positive integer, to achieve precise matching between the number of sub-viewpoints and the pixel array.

[0012] Optionally, the hyperspectral sensor is a snapshot hyperspectral image sensor, which is composed of a multispectral filter array and an image sensor, providing the acquisition model with a coupled acquisition function of spectral information and light field information; Multispectral filter array is composed of The spectral modulation material is arranged in a periodic cycle and is fixedly mounted on the surface of the photodiode array of the image sensor to realize the spectral domain encoding function of high-dimensional hyperspectral information of skin tissue. A complete spectral modulation unit is formed by a type of spectral modulation material, and the spectral modulation unit is respectively connected to the image sensor. The individual sub-microlenses of the pixel array and microlens array are matched one-to-one to achieve synchronous and coordinated spectral encoding and light field acquisition. The geometric dimensions of the image sensor are Image resolution is The physical size of a pixel is P, where L, M, and P are all positive real numbers. The image sensor, as the core acquisition element, is used to receive the light signals of skin tissue transmitted through the lens group, acquire the spectral and three-dimensional spatial coupling information of the skin tissue, and generate and output spectral coupled light field data based on the acquired light signals.

[0013] Optionally, the reconstructed hyperspectral light field image further includes: Construct a dataset linking spectral coupled light field images and hyperspectral light field images, and establish a mapping relationship between sampled data and hyperspectral data; Using the trained and optimized spectral reconstruction network, at a resolution of The spectral coupled light field data is processed pixel-by-pixel and reconstructed to finally obtain a resolution of The hyperspectral light field image, in which The number of spectral channels enables precise reconstruction from coupled data into a hyperspectral light field image.

[0014] Optionally, acquiring multi-view hyperspectral images from different perspectives also includes: According to a preset sampling rule, synchronous sampling is performed on different positions of all macro pixels in the reconstructed hyperspectral light field image, based on the acquisition model. Collect sub-view counts and extract those that match the collected sub-view counts. The multi-view hyperspectral images have a resolution of [missing information - likely a value or value]. ,in N is a positive integer, enabling accurate extraction of light field information from multiple perspectives.

[0015] Optionally, the process of utilizing optical field super-resolution processing includes: Multi-view hyperspectral images of each spectral band are input into a pre-trained optical field super-resolution network. N-fold super-resolution enhancement is performed on the multi-view hyperspectral images. Through feature extraction and pixel reconstruction functions of the optical field super-resolution network, high-resolution multi-view hyperspectral images are generated band by band. Finally, high-resolution multi-view hyperspectral images corresponding to each band are output, with a resolution of [resolution value missing]. This enables resolution improvement of multi-view light field images.

[0016] Optionally, the generation of superficial skin tissue depth maps and hyperspectral three-dimensional images also includes: The target bands with optimal skin tissue penetration were selected from the high-resolution multi-view hyperspectral images of each band. The corresponding high-resolution multi-view hyperspectral image, in the target band Using the central view image of the corresponding image as a reference, stereo matching calculations are performed on the other view images to obtain a high-precision depth map of the superficial skin tissue. High-resolution center-view images are extracted from the high-resolution multi-view hyperspectral images of each spectral channel. Spatial coordinate calibration and pixel-level fusion registration are performed on the high-resolution center-view images of each spectral channel and the skin superficial tissue depth map. Combining spectral information and three-dimensional spatial depth information, a hyperspectral three-dimensional image of the skin tissue is finally generated, realizing the integrated reconstruction of skin tissue spectral and three-dimensional information.

[0017] To achieve the above objectives, a second aspect of the present invention provides a snapshot-type hyperspectral three-dimensional skin imaging device, comprising: The data acquisition module is used to construct a snapshot-type spectral-light field coupled imaging system containing a light source model and an acquisition model. The light source model outputs broadband illumination to achieve full-area skin illumination, and the acquisition model works in conjunction with a hyperspectral sensor through a lens group to synchronously couple and acquire skin spectral and three-dimensional spatial information to generate spectral coupled light field data. The image reconstruction module is used to establish an associated dataset of spectral coupled light field images and hyperspectral light field images, train a spectral reconstruction network based on the associated dataset, solve the spectral coupled light field data and reconstruct the hyperspectral light field image through the spectral reconstruction network, and obtain multi-view hyperspectral images from different perspectives through macro-pixel multi-position sampling. The super-resolution enhancement module is used to perform super-resolution processing on multi-view hyperspectral images using an optical field super-resolution network to obtain high-resolution multi-view hyperspectral images corresponding to each spectral band, and select the high-resolution multi-view image corresponding to the optimal penetration band for stereo matching to generate a skin superficial tissue depth map. The 3D construction module is used to select the target band image with the best skin penetration from the high-resolution multi-view images of each spectral channel, perform stereo matching based on the center view image to obtain the target band depth map, and fuse the high-resolution center view images of each channel with the target band depth map to complete the generation of hyperspectral 3D stereo image.

[0018] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0019] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory to implement a snapshot-type hyperspectral three-dimensional skin imaging method as described in the first aspect embodiment.

[0020] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a snapshot-type hyperspectral three-dimensional skin imaging method as described in the first aspect embodiment.

[0021] The embodiments of the present invention have the following beneficial effects: 1. Relying on the collaborative architecture of microlens array and snapshot hyperspectral image sensor, the synchronous acquisition of spectral coupled light field data can be completed in a single exposure, eliminating the need for the multi-frame scanning imaging process of traditional hyperspectral dermoscopy, greatly shortening the imaging time and meeting the clinical diagnostic needs for rapid detection.

[0022] 2. Overcoming the limitations of traditional dermoscopy, which can only acquire two-dimensional surface images or single-spectral information, this technology utilizes hyperspectral reconstruction to obtain multi-band spectral features of skin tissue. Combined with depth maps generated through stereo matching, it achieves the fusion of hyperspectral and three-dimensional structural information. By leveraging the specific spectral responses of components such as melanin and collagen, as well as the deep infiltration characteristics of lesions, it can accurately distinguish between normal and malignant tissues, significantly improving the detection sensitivity and diagnostic accuracy of early-stage skin cancer and other lesions.

[0023] 3. The light source model supports two modes: white light single light source or white light-infrared light composite light source. White light can cover the visible light to near-infrared spectrum range to achieve full-area illumination of the skin surface; infrared light can supplement the spectral coverage range and enhance the penetration depth of skin tissue, which can meet the detection needs of different types and depths of skin lesions and expand the applicability of the equipment.

[0024] 4. By adopting a deep learning architecture that combines a spectral reconstruction network and a light field super-resolution network, high-resolution multi-view hyperspectral images can be efficiently reconstructed from the original coupled data. This also improves the spatial and spectral resolution of the images, providing high-quality data support for subsequent stereo matching and 3D imaging, and ensuring the clarity and accuracy of the final 3D stereo images. Attached Figure Description

[0025] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a snapshot-type hyperspectral three-dimensional skin imaging method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a direct-coupled optical path structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a relay imaging optical path structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram showing the positional correspondence between MLA and BMSFA provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the hyperspectral three-dimensional reconstruction method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the data processing flow provided in an embodiment of the present invention; Figure 7 This is a structural diagram of a snapshot-type hyperspectral three-dimensional skin imaging device provided in an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] A snapshot-type hyperspectral three-dimensional skin imaging method and apparatus according to an embodiment of the present invention will now be described with reference to the accompanying drawings.

[0029] Example 1 This invention provides a snapshot-based hyperspectral three-dimensional skin imaging method. Figure 1This is a schematic flowchart of a snapshot-type hyperspectral three-dimensional skin imaging method provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S1: Construct a snapshot-type spectral-light field coupling imaging system containing a light source model and an acquisition model. The light source model outputs broadband illumination to achieve full-area skin illumination. The acquisition model works in conjunction with a hyperspectral sensor through a lens group to synchronously couple and acquire skin spectral and three-dimensional spatial information, generating spectral-coupled light field data.

[0030] In this embodiment, the configuration of the light source model has been specifically optimized, employing a flexible scheme of white light illumination alone or white light combined with infrared light illumination to adapt to different detection scenarios for skin lesions. The white light source covers a broad spectral range from visible light to near-infrared (NIR), which precisely encompasses the characteristic absorption and scattering spectral bands of key biological components in skin tissue, such as melanin, collagen, and hemoglobin. Melanin exhibits strong absorption characteristics in the visible light band, collagen has a characteristic reflection peak in the near-infrared band, and hemoglobin shows a distinct absorption trough at specific wavelengths. This characteristic spectral information is crucial for distinguishing normal tissue from diseased tissue and determining the nature of lesions. Broad-spectrum white light illumination provides a sufficient and comprehensive spectral information foundation for subsequent spectral analysis and identification. The introduction of infrared light further expands the system's detection capabilities: on the one hand, infrared light can fill the coverage gap of white light in the long-wavelength band, making the spectral acquisition range more complete and capturing the spectral response of more deep skin tissues; on the other hand, infrared light has a stronger skin tissue penetration ability than visible light, and can penetrate the skin surface to reach the dermis and even the superficial subcutaneous tissue, thereby obtaining structural information inside the skin, which is of great significance for assessing the depth of lesion infiltration; at the same time, considering that the photoelectric conversion sensitivity of conventional image sensors is often reduced in the infrared band, the supplementary illumination of infrared light can effectively compensate for this deficiency, ensuring that the system can still obtain high-quality, high signal-to-noise ratio imaging data in the infrared band, and ensuring the stability of imaging quality across the entire spectrum.

[0031] As the core execution unit for system data acquisition, the acquisition model in this embodiment adopts a modular architecture in which the main lens, microlens array, and snapshot hyperspectral image sensor work in sequence. The parameter matching and optical path design between each component have undergone precise calculation and debugging. Among them, the microlens array is composed of multiple sub-microlenses of the same specifications arranged in a regular manner. Its overall array size strictly matches the effective photosensitive area size of the snapshot hyperspectral image sensor, ensuring that light can be fully utilized without waste. The microlens array is positioned between the main lens and the snapshot hyperspectral image sensor. Based on the space constraints and imaging quality requirements of the actual application scenario, two optional optical path structures are provided: The first is a direct-coupled optical path structure, where the microlens array is directly fixed to the surface of the snapshot hyperspectral image sensor, ensuring the sensor's photosensitive surface is precisely positioned at the focal plane of the microlens. This structure offers the advantage of a simple and compact optical path, eliminating the need for additional relay optical components, which facilitates system miniaturization and is suitable for the development of portable dermoscopy devices. The second is a relay imaging optical path structure, where a 4f relay imaging system is added between the microlens array and the sensor. Through optical control of the relay lens group, the focal plane of the microlens array and the photosensitive surface of the sensor form a conjugate optical relationship. This design effectively corrects aberrations in the optical path, improves light convergence efficiency and imaging clarity, and is particularly suitable for clinical diagnostic scenarios with extremely high imaging quality requirements. Meanwhile, to ensure efficient optical transmission and consistent imaging, the F-numbers of the microlenses are strictly kept consistent with those of the main lens to avoid light loss or imaging distortion caused by mismatched optical parameters. The spacing D of the microlens array is determined by the pixel size P of the snapshot hyperspectral image sensor and the subsequent viewing angle sampling requirements (i.e., the number of pixels N corresponding to a single microlens) using the formula... Precise calculations show that this design ensures that the light field information collected by each sub-microlens can be completely received by the corresponding pixel array of the sensor, achieving accurate mapping between viewpoint information and pixel data.

[0032] In this embodiment, the snapshot hyperspectral image sensor, as the core sensing component of the acquisition model, adopts an integrated design combining a multispectral filter array and an underlying image sensor, possessing dual functions of spectral encoding and light field acquisition. The multispectral filter array consists of... The spectral modulation materials are arranged in a cyclic pattern, and this array is tightly fixed to the surface of the photodiode array of the underlying image sensor, forming a seamless sensing unit. Notably, the N×N spectral modulation materials constitute an independent spectral modulation unit, which is spatially integrated with the underlying image sensor... Each pixel corresponds perfectly to a sub-microlens in the microlens array, forming a one-to-one optical relationship. This one-to-one correspondence design of "microlens-spectral modulation unit-pixel array" is key to achieving synchronous coupling and acquisition of spectral and optical field information: light rays from different directions captured by each sub-microlens are precisely incident on the corresponding spectral modulation unit, which encodes the incident light in the spectral domain, converting high-dimensional hyperspectral information into two-dimensional image data that can be captured by the pixel array. The underlying image sensor adopts a high-resolution design, with a size of [missing information - likely a unit size]. The resolution reached With a pixel size of P, it can accurately capture image data after spectral modulation and light field coupling, ensuring the high fidelity of the original data and providing a reliable data source for subsequent algorithm processing.

[0033] In actual clinical applications, the imaging process of this system is clear and efficient: First, the broadband light emitted by the light source model is converged and collimated by the main lens and then uniformly illuminates the skin target detection area. The absorption, reflection, and scattering characteristics of skin tissue to incident light vary depending on the tissue composition and structural morphology. These differences precisely carry information about the physiological structure and biochemical composition of the skin tissue. The reflected light carrying this information is converged again by the main lens and then incident on a microlens array, where multi-view separation is performed—each sub-microlens captures light from different directions, thus decomposing the light field information in three-dimensional space into multiple two-dimensional images from different perspectives. Subsequently, this information after perspective separation is incident on a snapshot hyperspectral image sensor, where it is spectrally encoded by a multispectral filtering array, converting the image from each perspective into an encoded image containing spectral information. Finally, the underlying image sensor performs photoelectric conversion, converting the optical signal into a digital signal, ultimately generating... Pixel-level spectral coupled light field data. Through this complete optical path transmission and data acquisition process, the embodiments of this application successfully achieved single-exposure synchronous acquisition of spectral information and three-dimensional spatial information, laying a solid hardware data foundation for subsequent hyperspectral reconstruction and three-dimensional reconstruction.

[0034] Step S2: Establish an associated dataset of spectral coupled light field image and hyperspectral light field image, train a spectral reconstruction network based on the associated dataset, solve the spectral coupled light field data and reconstruct the hyperspectral light field image through the spectral reconstruction network, and obtain hyperspectral images from different perspectives through macro-pixel multi-position sampling.

[0035] In this embodiment, the construction of the associated dataset is a core foundational step in the hyperspectral reconstruction process. The quality of this dataset directly determines the accuracy and fidelity of the subsequent hyperspectral light field image reconstruction results, and is a crucial prerequisite for ensuring the feasibility of the entire technical solution. The construction of this dataset strictly adheres to the principle of precise "input-output" correspondence. The input data consists of spectral-coupled light field images acquired by a snapshot hyperspectral image sensor in actual acquisition scenarios. This image data contains mixed information after spectral modulation and light field coupling, serving as the original data source for subsequent reconstruction calculations. The output data is a high-fidelity hyperspectral light field image corresponding to the input image. This image needs to be acquired synchronously using a high-precision spectral measurement device to ensure it accurately reflects the light field distribution characteristics of the target scene in each spectral channel. To establish a precise mapping relationship between the sampled data and the hyperspectral light field data, this embodiment collects scene data from a large number of different skin tissue samples, under different lighting conditions, and with different lesion types. Each set of input spectral-coupled light field images is annotated with corresponding hyperspectral light field images, forming a large-scale and comprehensive training sample library. This provides sufficient and diverse learning data support for the training of the spectral reconstruction network.

[0036] Based on the aforementioned constructed associated dataset, this application trains a dedicated spectral reconstruction network tailored to the characteristics of spectral coupled light field data. This network employs an end-to-end architecture design in deep learning, leveraging the synergistic effects of multi-layer convolutional neural networks, attention mechanisms, and feature fusion units to deeply mine the hidden spectral and spatial dimension correlation information within the spectral coupled light field data. Compared to traditional spectral reconstruction algorithms, this application's spectral reconstruction network eliminates the need for manually designed feature extraction rules. Through autonomous learning on massive datasets, it adaptively masters the encoding patterns of spectral coupled data, thereby achieving accurate inverse mapping from low-dimensional coupled data to high-dimensional hyperspectral light field data. This significantly improves reconstruction efficiency while maintaining high fidelity, meeting the real-time requirements of clinical diagnosis. In this embodiment, the data collected in step S1... Pixel-level spectral coupled light field data is input into a trained spectral reconstruction network. The network performs layer-by-layer computation on the input data through forward propagation, and the final output resolution remains constant. ,Include spectral channels ( This is a hyperspectral light field image (with a preset number of spectral channels based on clinical diagnostic needs, covering key characteristic bands from visible to near-infrared) with a size of [missing information]. The three-dimensional data matrix, where each dimension corresponds to the spatial horizontal coordinate, spatial vertical coordinate, and number of spectral channels, fully preserves the spatial structure information and spectral feature information of the target region.

[0037] After completing the hyperspectral light field image reconstruction, this embodiment further performs a macro-pixel multi-location sampling (i.e., pixel rearrangement) operation. The core purpose of this operation is to separate hyperspectral images from different perspectives from the hyperspectral light field data, providing data support for the subsequent stereo matching process of 3D reconstruction. In this embodiment, the pixel region corresponding to each sub-microlens on the sensor plane in the hyperspectral light field image is explicitly defined as a macro-pixel. Since each sub-microlens can capture information from different directions, the corresponding macro-pixel not only contains the two-dimensional structural information of the spatial position aligned by the sub-microlens, but also integrates all angle sampling information at that spatial position, realizing the fusion storage of spatial and angle information. According to the hardware parameter design of the acquisition model, each sub-microlens corresponds to the underlying image sensor. 1 pixel, therefore the size of each macro pixel is also set synchronously to 1. The macropixels are arranged in a regular array in the hyperspectral light field image. Based on this structural characteristic, this application can obtain the image from the image by regularly sampling the same positions of all macropixels (such as the top left and top right pixels of each macropixel). N×N hyperspectral images from different viewpoints are extracted from the standard hyperspectral light field image, and finally a size of [size missing] is formed. Multi-view hyperspectral image data, among which , representing the resolution of the hyperspectral image from a single viewpoint. This application, through this pixel rearrangement step, successfully transforms hyperspectral light field data, which integrates spatial, spectral, and angular information, into multi-view hyperspectral image data with a clear structure that can be directly used for stereo matching calculations. This lays a solid data foundation for the subsequent analysis of the correspondence between images from different viewpoints during 3D reconstruction, ensuring the accuracy and efficiency of stereo matching.

[0038] Step S3: Use a light field super-resolution network to perform super-resolution processing on the multi-view hyperspectral images to obtain high-resolution multi-view hyperspectral images corresponding to each spectral band. Then, select the high-resolution multi-view image corresponding to the optimal penetration band for stereo matching to generate a skin superficial tissue depth map.

[0039] In this embodiment, although the pixel rearrangement operation performed in step S2 successfully separated the multi-view hyperspectral image, this operation reduced the original resolution to [missing information]. Hyperspectral light field images, according to the correspondence of individual sub-microlenses The rule of each pixel is broken down into... Sub-images from multiple viewpoints cause the resolution of the image at each viewpoint to decrease to a certain value. (in The reduction in resolution directly affects the matching accuracy of pixel-level features in subsequent stereo matching processes. Low-resolution images struggle to clearly depict the subtle structures of skin lesions (such as the morphology of small blood vessels and differences in cell arrangement), potentially leading to feature point matching errors or disparity calculation deviations, thus affecting the accuracy of depth information. Therefore, to ensure the final effect of 3D reconstruction, it is necessary to enhance the resolution of multi-view hyperspectral images through super-resolution processing to restore the image's detailed information.

[0040] The light field super-resolution network used in this application is a deep learning network specifically designed based on the characteristics of multi-view light field images. Compared with traditional single-frame image super-resolution algorithms, its core advantage lies in its ability to fully mine and utilize the spatial correlation and complementary information between images from different viewpoints. This network, by introducing a multi-view feature alignment module, a cross-view attention fusion unit, and a high-dimensional feature reconstruction layer, improves image spatial resolution while strictly maintaining the geometric consistency and spectral information fidelity between images from different viewpoints—avoiding viewpoint shifts or spectral feature distortions caused by super-resolution processing, and ensuring that the correspondence between images from different viewpoints is not affected during subsequent stereo matching. In this application embodiment, the size is... After multi-view hyperspectral images are input into this optical field super-resolution network, the network first extracts shallow texture features and deep structural features from each viewpoint through convolutional layers. Then, a cross-view attention fusion unit performs correlation calculations on the feature maps from different views, selects complementary feature information, and fuses them to enhance the feature expression of the lesion region. Finally, a high-dimensional feature reconstruction layer upsamples and restores details of the fused feature map, completing N-fold super-resolution enhancement. The final output is a high-resolution multi-view hyperspectral image corresponding to each band, with a size of [size missing]. This allows the resolution of each viewpoint image to be fully restored to the highest resolution level of the original snapshot hyperspectral image sensor, providing high-definition, high-fidelity image data support for subsequent stereo matching.

[0041] After completing the super-resolution processing, this embodiment of the application further selects the target band with optimal skin penetration from the high-resolution multi-view hyperspectral images of each spectral band. The corresponding image selection process is a crucial step in ensuring the effectiveness of the depth information in the 3D reconstruction. In this embodiment, the propagation characteristics of light in skin tissue differ significantly across spectral bands. Visible light is primarily absorbed and scattered by the stratum corneum and melanocytes on the skin's surface, resulting in shallow penetration and only reflecting the surface and superficial structural information. In contrast, infrared or near-infrared light has a lower absorption coefficient and weaker scattering effect, allowing it to penetrate to the dermis and even superficial subcutaneous tissue, thus obtaining information on the internal layer structure and lesion infiltration of the skin. Therefore, this application precisely selects the optimal penetration band by quantitatively analyzing the skin penetration depth of each spectral band (combining optical property parameters of skin tissue with experimental measurement data). As a reference band for three-dimensional reconstruction, it ensures that the depth map obtained by subsequent stereo matching can truly reflect the internal structure of the superficial skin tissue, providing a reliable basis for assessing the depth of lesion infiltration.

[0042] Subsequently, embodiments of this application use the target band Using the center view image in the corresponding high-resolution multi-view image as a reference, stereo matching calculation is performed, which is the core step in realizing 3D reconstruction. In this embodiment, the core logic of stereo matching is based on the principle of triangulation. By calculating the disparity of corresponding pixels in images from different viewpoints, the depth information is derived. Specifically, the center view image and other viewpoint images are first preprocessed, including image denoising, edge enhancement, and feature point extraction (such as using algorithms like SIFT and SURF to extract feature points with rotation invariance and scale invariance). Then, through block matching or feature point matching algorithms, the differences in grayscale distribution and texture structure of feature points in the center view image and other viewpoint images are compared to determine the pixel coordinate offset of the corresponding feature points under different viewpoints, i.e., disparity. Next, according to the preset geometric constraints (including hardware parameters such as the spacing of the microlens array, the focal length of the main lens, and the relative position of the sensor and the microlens array), a mapping model between disparity and depth is established, and the calculated disparity is converted into the corresponding physical depth value. Finally, through interpolation filling and smoothing, noise and holes in the depth map are eliminated to generate a continuous depth map that clearly reflects the 3D structure of the superficial skin tissue. This depth map accurately records the depth information of various points on and inside the skin surface, providing crucial depth data support for the subsequent construction of hyperspectral three-dimensional images, ensuring that the final generated three-dimensional image can realistically reproduce the spatial structure and morphology of the skin tissue.

[0043] Step S4: Select the target band image with the best skin penetration from the high-resolution multi-view images of each spectral channel, perform stereo matching based on the center view image to obtain the target band depth map, and fuse the high-resolution center view images of each channel with the target band depth map to complete the generation of the hyperspectral three-dimensional image.

[0044] In this embodiment, to further ensure the reliability of depth information, the accuracy of the depth map generated in step S3 needs to be verified by combining the features of high-resolution multi-view images from each spectral channel. Since images from different spectral channels have complementary representations of skin tissue structure—for example, some spectral channels significantly enhance the contrast of skin blood vessels, while others more clearly delineate the boundary between diseased and normal tissue—this application uses cross-spectral channel feature consistency verification to determine whether the depth value of each pixel in the depth map matches the tissue structure features at the corresponding location. Specifically, if a location exhibits a clear layered structure in images from multiple spectral channels, but the depth value of that region in the depth map lacks gradient changes, then the depth information in that region is deemed abnormal and needs to be optimized through interpolation correction or re-matching. Through this verification process, it is ensured that the depth map can realistically and accurately reflect the actual three-dimensional structure of skin tissue, laying a reliable foundation for subsequent data fusion.

[0045] After verifying and optimizing the depth map, this embodiment of the application fuses and matches the high-resolution center-view images of each spectral channel with the optimized depth map. The core of this process is to establish a precise spatial correspondence between spectral information and depth information, thereby achieving the organic integration of multi-dimensional data. In this embodiment, the fusion matching first eliminates system errors through coordinate calibration technology: Although the acquisition paths of the spectral image and the depth map are based on the same hardware system, slight spatial misalignment may exist due to factors such as the installation accuracy of optical components and algorithm calculation deviations. Therefore, it is necessary to use the feature markers on the skin surface (such as the feature inflection points of lesion edges, blood vessel intersections, etc.) as a reference to align the spatial coordinates of the high-resolution center-view images of each spectral channel with the depth map, ensuring that the spatial position of each pixel in the spectral image completely matches the corresponding depth value in the depth map, avoiding phenomena such as offset, stretching, or distortion.

[0046] After coordinate calibration, this application employs a multi-dimensional data fusion algorithm to deeply fuse the spectral feature information and depth structure information of each spectral channel. This algorithm constructs a three-dimensional data tensor model, using the pixel grayscale value (representing spectral features) of the high-resolution center view image under each spectral channel as the spectral dimension parameter of the tensor, and the depth value corresponding to the depth map as the spatial depth dimension parameter of the tensor, while retaining the two-dimensional spatial coordinate information of the pixels. The final result is a tensor containing spatial dimensions (horizontal and vertical) and spectral dimensions (…). The algorithm combines hyperspectral three-dimensional image data (including multiple channels and depth dimension) with high-spectral 3D image data. During the fusion process, the algorithm uses a weight allocation mechanism to highlight key information—assigning higher weights to spectral feature bands that are important for lesion diagnosis and regions with significant depth gradient changes, ensuring that the fused image clearly presents the core diagnostic information.

[0047] In this embodiment, the generated hyperspectral three-dimensional image possesses significant technical advantages: on the one hand, it can clearly reproduce the fine surface structure of skin tissue (such as the morphology of scales and pigmentation spots) and internal layer structure (such as the distribution of blood vessels in the dermis and the infiltration range of lesions), providing doctors with an intuitive three-dimensional structural reference; on the other hand, the multispectral feature information contained in the image can accurately reflect the component differences in different tissue regions, such as the difference in grayscale values ​​between malignant lesions and normal tissues in specific spectral channels, helping doctors quickly identify the extent and nature of lesions. Based on this hyperspectral three-dimensional image, doctors can comprehensively grasp the morphology, size, boundary features, infiltration depth, and spectral characteristics of skin lesions, providing comprehensive and accurate reference for the qualitative diagnosis (benign / malignant), malignancy assessment (such as Clark classification of melanoma), and personalized treatment plan formulation (such as surgical resection range planning) of skin lesions, effectively reducing the risk of misdiagnosis and missed diagnosis.

[0048] Through the coordinated operation of steps S1 to S4, this embodiment of the application successfully achieved single-exposure hyperspectral three-dimensional imaging of superficial skin tissues: the hardware system in step S1 ensured the synchronous acquisition of spectral and spatial information; the spectral reconstruction and pixel rearrangement in step S2 achieved effective separation of multi-view data; the super-resolution processing and stereo matching in step S3 ensured the accuracy of image quality and depth information; and the data fusion in step S4 formed a multi-dimensional diagnostic image. The entire technical solution overcomes the core shortcomings of traditional dermatoscopes, which can only provide two-dimensional surface images and lack depth and spectral information, and also solves the problems of traditional hyperspectral imaging requiring multiple frame scans and having slow imaging speed. It significantly improves the accuracy and efficiency of skin lesion diagnosis, providing an innovative and practical technical means for clinical dermatological diagnosis, and has significant clinical application value and promising prospects for promotion.

[0049] In the application of one embodiment of the present invention, the implementation process is as follows: Figure 2 and Figure 3 Schematic diagrams of two optical path structures to realize the present invention. Figure 2 The microlens array is directly fixed to the surface of the snapshot hyperspectral image sensor, so that the sensor is located at the focal plane of the microlens; Figure 3A 4f system is added between the microlens array and the snapshot hyperspectral image sensor for relay imaging, making the focal plane of the microlens array and the photosensitive surface of the sensor conjugate surfaces. The snapshot spectral coupling light field imaging system of this invention consists of two parts: a light source model and an acquisition model, completing single-exposure coupled acquisition of spectral and three-dimensional information. The system's light source model comprises six white LEDs and six infrared LEDs. Based on different peak wavelengths, the infrared LEDs are further divided into three groups (wavelengths of 750nm, 850nm, and 950nm), with the twelve LEDs arranged in a ring. The light emitted by the LEDs is directed towards the skin surface; some light is absorbed by the skin, and light of different wavelengths penetrates to different depths according to the absorption and scattering characteristics of tissue components. Unabsorbed light is reflected or transmitted back to the sensor array of the dermoscope at different angles, recording spectral characteristics and spatial distribution information.

[0050] The system's acquisition model consists of a main lens, a microlens array, and a snapshot hyperspectral image sensor. The snapshot hyperspectral image sensor comprises two components: a broadband multispectral filter array (BMSFA) and a broadband monochrome image sensor. The main lens has an F-number of 2 and a diameter of 20mm. The BMSFA is composed of 5×5 broadband spectral modulation materials arranged in a circular pattern, and is bonded to the surface of the broadband monochrome image sensor's photodiode array using SU-8 photoresist. Each 5×5 broadband spectral modulation material represents a broadband spectral modulation unit and corresponds to 5×5 pixels of the underlying sensor. Furthermore, the broadband monochrome image sensor measures 5.12mm × 5.12mm, has a resolution of 2048 × 2048, and a pixel size of 5μm.

[0051] The microlens array measures 6mm × 6mm, and the number of sub-viewpoints is 5 × 5, meaning each circular microlens covers 5 × 5 sensor pixels, corresponding to 25 different viewpoints. For example... Figure 4 As shown, each microlens also corresponds to a broadband spectral modulation unit, i.e., a 5×5 broadband spectral modulation material. To match the microlens aperture to the 5×5 sensor pixels, the spacing D of the microlens array can be calculated to be 25 μm based on the sensor pixel size. Furthermore, to maximize the utilization of the detector pixels, the F-number of the microlenses is kept consistent with the F-number of the main lens, both being 2. The focal length of the microlenses can be determined using the following formula. It is 50μm.

[0052]

[0053] like Figure 5 The diagram shown is a flowchart of the hyperspectral three-dimensional reconstruction method according to an embodiment of the present invention. The specific implementation steps are as follows: Step 1: The snapshot-type hyperspectral image sensor acquired 2048×2048 pixel spectral coupled light field data. A dataset of spectral coupled light field images and hyperspectral light field images was constructed, and the spectral reconstruction network SRNet was selected for training. The spectral coupled light field data (2048×2048) was input into the trained SRNet network model for hyperspectral reconstruction, generating 61 channels in the range of 400~1000nm, with an interval of 10nm. This resulted in hyperspectral light field data (2048×2048×61), with each channel having a resolution of 2048×2048.

[0054] Step 2: Rearrange the pixels of the reconstructed hyperspectral light field data (2048×2048×61) to obtain 5×5 hyperspectral images (409×409×61×5×5) from different viewpoints. For example... Figure 6 This is a schematic diagram of the data processing flow of the present invention, in which pixel rearrangement converts hyperspectral light field data into 5×5 hyperspectral images with different viewing angles. The 5×5 different pixels behind each microlens correspond to different angular coordinates. Different perspectives, with the central perspective being .

[0055] Step 3: Use the optical field super-resolution network Vs-Net to perform 5x super-resolution on the 5×5 view images (409×409×61×5×5) of each band, and reconstruct the high-resolution 5×5 view images (2045×2045×61×5×5) corresponding to each band.

[0056] Step 4: Select an infrared image with a wavelength of 1000nm from 61 high-resolution multi-view images (2045×2045×61×5×5), and perform stereo matching using the center viewpoint as the reference image. By comparing the pixel positions of the same object in different views of this band, parallax is calculated, and depth information is derived to obtain a shallow tissue depth map. The specific process of calculating depth is as follows.

[0057] Assumption Indicates the first One perspective The pixel intensity value at that location, where It is a view index, corresponding to the center view. , These are the spatial coordinates of a pixel within the image. For each pixel... parallax Cost function Taking into account the influence of different perspectives, the degree of fit between the central perspective and other perspectives is measured. For each non-central perspective... The pixel matching error can be expressed as:

[0058] in and It is parallax Projections in the horizontal and vertical directions, and This is the baseline scaling factor between viewpoints. By traversing and integrating different viewpoints, the overall cost function is obtained:

[0059] To solve for the optimal disparity We need to minimize the cost function. By iterating through all possible disparity values Calculate each The cost is calculated, and the minimum cost is selected. To improve matching efficiency, a block matching algorithm is used to divide the image into small blocks. For each block, the cost function under different disparities is calculated, and the disparity corresponding to the minimum cost is selected. By repeating the above process, the disparity map of the entire image can be obtained.

[0060]

[0061] Furthermore, based on baseline distance and focal length Depth can be calculated using the following formula. Thus, the depth map is obtained from the disparity map.

[0062]

[0063] After obtaining the depth map, the high-resolution center-view images of each band are finally matched with the depth map to obtain a hyperspectral three-dimensional image, thus realizing single-exposure hyperspectral three-dimensional imaging.

[0064] Example 2 This invention provides a snapshot-type hyperspectral three-dimensional skin imaging device. Figure 7 This is a schematic flowchart of a snapshot-type hyperspectral three-dimensional skin imaging device provided in an embodiment of the present invention. Figure 7 As shown, the device includes: The data acquisition module 100 is used to construct a snapshot-type spectral-light field coupled imaging system containing a light source model and an acquisition model. The light source model outputs broadband illumination to achieve full-area skin illumination. The acquisition model works in conjunction with a hyperspectral sensor through a lens group to synchronously couple and acquire skin spectral and three-dimensional spatial information to generate spectral coupled light field data. The image reconstruction module 200 is used to establish an associated dataset of spectral coupled light field image and hyperspectral light field image, train a spectral reconstruction network based on the associated dataset, solve the spectral coupled light field data and reconstruct the hyperspectral light field image through the spectral reconstruction network, and obtain multi-view hyperspectral images from different perspectives through macro-pixel multi-position sampling. The super-resolution enhancement module 300 is used to perform super-resolution processing on multi-view hyperspectral images using an optical field super-resolution network to obtain high-resolution multi-view hyperspectral images corresponding to each spectral band, and select the high-resolution multi-view image corresponding to the optimal penetration band for stereo matching to generate a skin superficial tissue depth map. The 3D construction module 400 is used to select the target band image with the best skin penetration from the high-resolution multi-view images of each spectral channel, perform stereo matching based on the center view image to obtain the target band depth map, and fuse the high-resolution center view images of each channel with the target band depth map to complete the generation of hyperspectral 3D stereo images.

[0065] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0066] Example 3 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0067] Example 4 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0069] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0070] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A snapshot-type hyperspectral three-dimensional skin imaging method, characterized in that, include: S1. Construct a snapshot-type spectral-light field coupled imaging system containing a light source model and an acquisition model. The light source model outputs broadband illumination to achieve full-area skin illumination. The acquisition model works in conjunction with a hyperspectral sensor through a lens group to synchronously couple and acquire skin spectral and three-dimensional spatial information to generate spectral coupled light field data. S2, establish a correlation dataset between the spectral coupled light field image and the hyperspectral light field image, train a spectral reconstruction network based on the correlation dataset, solve the spectral coupled light field data and reconstruct the hyperspectral light field image through the spectral reconstruction network, and obtain multi-view hyperspectral images from different perspectives through macro-pixel multi-position sampling. S3 uses a light field super-resolution network to perform super-resolution processing on multi-view hyperspectral images to obtain high-resolution multi-view hyperspectral images corresponding to each spectral band, and selects the high-resolution multi-view image corresponding to the optimal penetration band for stereo matching to generate a skin superficial tissue depth map. S4. Select the target band image with the best skin penetration from the high-resolution multi-view images of each spectral channel, perform stereo matching based on the center view image to obtain the target band depth map, and fuse the high-resolution center view images of each channel with the target band depth map to complete the generation of the hyperspectral three-dimensional image.

2. The method according to claim 1, characterized in that, The lens group includes a main lens and a microlens array. The microlens array is composed of multiple sub-microlenses with identical structures arranged in a periodic array. The microlens array is fixedly set in the imaging optical path between the main lens and the hyperspectral sensor, providing multi-view light field information acquisition function for the acquisition model. There are two switchable optical path assembly structures between the microlens array and the hyperspectral sensor. One is to directly attach the microlens array to the photosensitive surface of the hyperspectral sensor, so that the photosensitive surface of the hyperspectral sensor is precisely located at the focal plane of the microlens array. The other is to connect a relay imaging system between the microlens array and the hyperspectral sensor, so that the focal plane of the microlens array and the photosensitive surface of the hyperspectral sensor form a conjugate surface, ensuring the distortion-free transmission of light field information. The number of sub-views acquired is determined by the sensor pixel array size corresponding to a single sub-microlens. When using a pixel array, the number of sub-viewpoints acquired is , where N is a positive integer, to achieve precise matching between the number of sub-viewpoints and the pixel array.

3. The method according to claim 2, characterized in that, The hyperspectral sensor is a snapshot hyperspectral image sensor, which is composed of a multispectral filter array and an image sensor, providing the acquisition model with a coupled acquisition function of spectral information and light field information; Multispectral filter array is composed of The spectral modulation material is arranged in a periodic cycle and is fixedly mounted on the surface of the photodiode array of the image sensor to realize the spectral domain encoding function of high-dimensional hyperspectral information of skin tissue. A complete spectral modulation unit is formed by a type of spectral modulation material, and the spectral modulation unit is respectively connected to the image sensor. The individual sub-microlenses of the pixel array and microlens array are matched one-to-one to achieve synchronous and coordinated spectral encoding and light field acquisition. The geometric dimensions of the image sensor are Image resolution is The physical size of a pixel is P, where L, M, and P are all positive real numbers. The image sensor, as the core acquisition element, is used to receive the light signals of skin tissue transmitted through the lens group, acquire the spectral and three-dimensional spatial coupling information of the skin tissue, and generate and output spectral coupled light field data based on the acquired light signals.

4. The method according to claim 3, characterized in that, The reconstructed hyperspectral light field image also includes: Construct a dataset linking spectral coupled light field images and hyperspectral light field images, and establish a mapping relationship between sampled data and hyperspectral data; Using the trained and optimized spectral reconstruction network, at a resolution of The spectral coupled light field data is processed pixel-by-pixel and reconstructed to finally obtain a resolution of The hyperspectral light field image, in which The number of spectral channels enables precise reconstruction from coupled data into a hyperspectral light field image.

5. The method according to claim 4, characterized in that, Acquiring multi-view hyperspectral images from different perspectives also includes: According to a preset sampling rule, synchronous sampling is performed on different positions of all macro pixels in the reconstructed hyperspectral light field image, based on the acquisition model. Collect sub-view counts and extract those that match the collected sub-view counts. The multi-view hyperspectral images have a resolution of [missing information - likely a value or value]. ,in N is a positive integer, enabling accurate extraction of light field information from multiple perspectives.

6. The method according to claim 5, characterized in that, The process of using light field super-resolution processing includes: Multi-view hyperspectral images of each spectral band are input into a pre-trained optical field super-resolution network. N-fold super-resolution enhancement is performed on the multi-view hyperspectral images. Through feature extraction and pixel reconstruction functions of the optical field super-resolution network, high-resolution multi-view hyperspectral images are generated band by band. Finally, high-resolution multi-view hyperspectral images corresponding to each band are output, with a resolution of [resolution value missing]. This enables resolution improvement of multi-view light field images.

7. The method according to claim 6, characterized in that, The generation of superficial skin tissue depth maps and hyperspectral three-dimensional images also includes: The target bands with optimal skin tissue penetration were selected from the high-resolution multi-view hyperspectral images of each band. The corresponding high-resolution multi-view hyperspectral image, in the target band Using the central view image of the corresponding image as a reference, stereo matching calculations are performed on the other view images to obtain a high-precision depth map of the superficial skin tissue. High-resolution center-view images are extracted from the high-resolution multi-view hyperspectral images of each spectral channel. Spatial coordinate calibration and pixel-level fusion registration are performed on the high-resolution center-view images of each spectral channel and the skin superficial tissue depth map. Combining spectral information and three-dimensional spatial depth information, a hyperspectral three-dimensional image of the skin tissue is finally generated, realizing the integrated reconstruction of skin tissue spectral and three-dimensional information.

8. A snapshot-type hyperspectral three-dimensional skin imaging device, characterized in that, include: The data acquisition module is used to construct a snapshot-type spectral-light field coupled imaging system containing a light source model and an acquisition model. The light source model outputs broadband illumination to achieve full-area skin illumination, and the acquisition model works in conjunction with a hyperspectral sensor through a lens group to synchronously couple and acquire skin spectral and three-dimensional spatial information to generate spectral coupled light field data. The image reconstruction module is used to establish an associated dataset of spectral coupled light field images and hyperspectral light field images, train a spectral reconstruction network based on the associated dataset, solve the spectral coupled light field data and reconstruct the hyperspectral light field image through the spectral reconstruction network, and obtain multi-view hyperspectral images from different perspectives through macro-pixel multi-position sampling. The super-resolution enhancement module is used to perform super-resolution processing on multi-view hyperspectral images using an optical field super-resolution network to obtain high-resolution multi-view hyperspectral images corresponding to each spectral band, and select the high-resolution multi-view image corresponding to the optimal penetration band for stereo matching to generate a skin superficial tissue depth map. The 3D construction module is used to select the target band image with the best skin penetration from the high-resolution multi-view images of each spectral channel, perform stereo matching based on the center view image to obtain the target band depth map, and fuse the high-resolution center view images of each channel with the target band depth map to complete the generation of hyperspectral 3D stereo image.

9. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images