Visible-near infrared hyperspectral imaging-based organ quality evaluation system and method
By using a visible-near-infrared hyperspectral imaging system and a machine learning model, the problem of non-destructive, real-time assessment of organ quality has been solved, achieving efficient and accurate organ quality assessment, which is suitable for organ transplantation assessment.
Patent Information
- Application Number
- CN202511168795.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-20
- Publication Date
- 2025-11-14
AI Technical Summary
Current technologies cannot achieve rapid, non-invasive, and real-time assessment of donor organ quality and function, resulting in uncertainty in organ transplant success rates and recipient survival rates.
An organ quality assessment system based on visible-near-infrared hyperspectral imaging is adopted, which uses linear gradient filters and imaging cameras combined with machine learning models to achieve non-destructive, high-throughput, and real-time assessment of organs.
It provides molecular fingerprint-level data on organ quality with an assessment accuracy of 97%, enabling rapid, non-destructive, and low-cost organ quality assessment, and is stable and portable.
Smart Images
Figure CN120938348A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intraoperative human organ quality assessment technology, and in particular to an organ quality assessment system and method based on visible-near-infrared hyperspectral imaging. Background Technology
[0002] Organ transplantation is the most effective treatment for end-stage renal and liver diseases, significantly improving patients' quality of life and survival. However, rapid and non-invasive assessment of the donor organ's quality and function is crucial for successful organ transplantation, directly determining the survival rate of the transplanted organ, the recipient's survival rate, and long-term prognosis.
[0003] Currently, clinical assessment of donor organ quality primarily relies on the transplant surgeon's clinical experience, pathological examination, clinical testing, and preoperative evaluation. However, the surgeon's experience and preoperative evaluation are not the most reliable methods, and pathological examination and clinical testing cannot provide non-invasive, real-time assessment results. Therefore, exploring a rapid, non-invasive organ quality and function assessment system and method is of significant clinical importance and value for organ transplantation. Summary of the Invention
[0004] To address the aforementioned issues, this invention provides an organ quality assessment system and method based on visible-near-infrared hyperspectral imaging, which enables rapid and non-destructive assessment of transplanted organs.
[0005] This invention is achieved through the following technical solution: In a first aspect, the present invention provides an organ quality assessment system based on visible-near-infrared hyperspectral imaging, comprising an imaging lens, a linear gradient filter, a convex lens and an imaging camera arranged in sequence along the optical axis, as well as a drive module and a host computer control module. The linear gradient filter is divided into n filtering regions in a preset direction. The width of each filtering region is denoted as s / n, and the center wavelengths of the light transmitted through each filtering region from one end to the other are denoted as λ1, λ2, ..., λ. n ,λ1,λ2……λ n The wavelength range of the linearly graded filter increases sequentially from 400 nm to 1000 nm; the preset direction is perpendicular to the optical axis. The driving module is used to drive the linear gradient filter to move along the preset direction, so that the filter area with the center wavelength of the transmitted light λ1 moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n. The imaging camera is used to acquire an image of the ex vivo organ after each step of the linear gradient filter moves, thus obtaining an image sequence. The host computer control module is used to extract valid images p1, p2, ..., p from the image sequence.m Calculate the number of pixel columns corresponding to each filtered region, denoted as t; extract columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on until image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral image; the single-pixel spectrum of each pixel is reconstructed from the hyperspectral image, and the obtained single-pixel spectra are classified and identified to obtain the organ quality assessment results.
[0006] Preferably, the driving module includes an axial displacement stage, an electrically controlled displacement stage, and a controller; a linear gradient filter is disposed on the axial displacement stage; the axial displacement stage is disposed on the electrically controlled displacement stage, and the axial displacement stage can move along the optical axis direction, so that the linear gradient filter coincides with the focal plane of the first image formed by the imaging lens after the ex vivo organ passes through the lens; The controller is used to drive the electronically controlled displacement stage to move, thereby moving the linear gradient filter along the preset direction, so that the filter area with the center wavelength of the transmitted light being λ1 moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n.
[0007] Preferably, the imaging camera has a spectral response range of 400 nm to 1000 nm.
[0008] Preferably, the host computer control module includes: an image extraction module, an image reconstruction module, and an evaluation model; The image extraction module is used to extract valid images p1, p2, ..., p from the image sequence output by the imaging camera. m Calculate the number of pixel columns corresponding to each filter region of the linear gradient filter, denoted as t; The image reconstruction module is used to reconstruct columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on up to image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral image; from λ2 to λ n The hyperspectral image is used to reconstruct the single-pixel spectrum of each pixel, and the obtained single-pixel spectrum is normalized and corrected to obtain the corrected single-pixel spectrum. The evaluation model is used to classify and identify the obtained corrected single-pixel spectra to obtain organ quality assessment results.
[0009] Furthermore, the evaluation model includes: a preprocessing module, a one-dimensional convolutional neural network and a long short-term memory network, and a classification module; The preprocessing module is used to sequentially perform light source calibration, spectral dimension reconstruction and data enhancement processing on the corrected single-pixel spectrum output by the image reconstruction module to obtain the preprocessed single-pixel spectrum. The one-dimensional convolutional neural network is used to extract local features from the preprocessed single-pixel spectrum to obtain spectral data with local features. The long short-term memory network is used to extract global features from spectral data with local features, thereby obtaining spectral data with global features. The classification module is used to classify organ quality based on the spectral data with global features.
[0010] Preferably, the one-dimensional convolutional neural network includes four convolutional layers, with the second and third convolutional layers being dilated convolutional layers; the output of each convolutional layer is connected to a batch normalization layer and a ReLU activation function; the output of the last convolutional layer is connected to a global max pooling layer; and the output of the global max pooling layer is connected to a residual connection layer. The convolutional layer is used to perform convolution operations on the preprocessed single-pixel spectrum to obtain the convolution result; The global max pooling layer is used to perform global max pooling on the convolution result to obtain the pooling result; The residual connection layer is used to perform residual connections on the pooling results to obtain spectral data with local features.
[0011] Preferably, the classification module includes: a global average pooling layer and a fully connected layer; A global average pooling layer is used to perform global average pooling on the spectral data with global features to obtain a feature vector of fixed length. The fully connected layer is used to calculate the classification probability using the Softmax function based on the fixed-length feature vector, and output the organ quality assessment result.
[0012] Secondly, the present invention provides an organ quality assessment method based on visible-near-infrared hyperspectral imaging, characterized in that the organ quality assessment system based on visible-near-infrared hyperspectral imaging comprises: Place the excised organ in front of the imaging lens and adjust the position of the linear graduated filter so that the linear graduated filter coincides with the focal plane of the first image formed by the excised organ after passing through the imaging lens. The linear gradient filter is driven by the driving module to move along the preset direction, so that the filtering area of the light transmitted by the linear gradient filter with a center wavelength of λ1 moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n; after each step of the linear gradient filter moves, an image of the ex vivo organ is acquired by the imaging camera to obtain an image sequence. The host computer control module extracts valid images p1, p2, ..., p from the image sequence. m Calculate the number of pixel columns corresponding to each filtering region of the linear gradient filter, denoted as t; calculate columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on until image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral image is used to reconstruct the single-pixel spectrum of each pixel; the obtained single-pixel spectra are then classified and identified to obtain the organ quality assessment results.
[0013] Preferably, the step of classifying and identifying the obtained single-pixel spectrum to obtain organ quality assessment results includes: The single-pixel spectrum is normalized, calibrated, reconstructed in spectral dimensions, and augmented with data to obtain the preprocessed single-pixel spectrum. Local features are extracted from the preprocessed single-pixel spectrum to obtain spectral data with local features; Global feature extraction is performed on spectral data with local features to obtain spectral data with global features; Based on the spectral data with global characteristics, organ quality is classified.
[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention employs a linear filter control module consisting of a continuously gradient filter capable of filtering visible to near-infrared light from 400 nm to 1000 nm and a driving module. This module filters out reflected light from different wavelengths of the target object (organ), obtaining the reflectance spectra of biological tissues such as hemoglobin and fat, providing molecular fingerprint-level data for organ quality assessment. An imaging camera and a host computer control module constitute an imaging processing module. The imaging camera acquires image data of ex vivo organs at different wavelengths, and the host computer control module performs spectral correction, image reconstruction, organ quality assessment, and result display. The hyperspectral imaging technology used in this invention offers advantages in organ quality assessment, including non-destructive, high-throughput, multi-parameter, and real-time detection, providing detailed information unavailable through traditional imaging and pathological methods.
[0015] Furthermore, the host computer control module embeds a machine learning model as an evaluation model for organ quality assessment, with an accuracy rate of up to 97%. It can achieve rapid and non-destructive assessment of ex vivo organs, and has the advantages of low cost, stability, portability, and speed. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of the organ quality assessment system based on visible-near-infrared hyperspectral imaging provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a linear gradient filter provided in an embodiment of the present invention; Figure 3 A flowchart of the single-pixel spectral normalization process provided in an embodiment of the present invention; Figure 4 This is a flowchart of the hyperspectral image reconstruction process provided in an embodiment of the present invention; Figure 5 A flowchart of the organ quality assessment algorithm provided in an embodiment of the present invention. Detailed Implementation
[0018] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.
[0019] It should be noted that the process equipment or apparatus not specifically mentioned in the following embodiments are all conventional equipment or apparatus in the art.
[0020] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Furthermore, unless otherwise stated, the numbering of each method step is merely a convenient tool for identifying each method step, and not intended to limit the order of the method steps or define the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.
[0021] refer to Figure 1This invention provides a rapid and non-destructive organ quality assessment system based on visible-near-infrared hyperspectral imaging. The system includes an optical zoom modulation module 100, a filter control module 200, and an imaging processing module 300.
[0022] The optical zoom modulation module 100 consists of a 4f system comprising an imaging lens 2 positioned at the front end of the linear graduated filter 4 and a convex lens 3 positioned at the rear end. The imaging lens 2, the linear graduated filter 4, and the convex lens 3 are arranged sequentially along the optical axis.
[0023] Specifically, the imaging lens 2 and the convex lens 3 are used together to adjust the imaging distance and image sharpness. In this embodiment, the imaging lens 2 is not limited and can be replaced according to the needs of the surgical site; correspondingly, the convex lens 3 also needs to be replaced. The imaging lens consists of a lens group.
[0024] The filter control module 200 consists of a linear gradient filter 4 and a drive module. The drive module consists of an axial displacement stage 5, an electrically controlled displacement stage 6, and a controller 7.
[0025] Specifically, such as Figure 2 As shown, the linear graded filter 4 is rectangular, but can also be of other shapes. Its wavelength range is from 400 nm to 1000 nm, and its spectral resolution is 6 nm. The linear graded filter can achieve continuous filtering of visible light to near-infrared light from 400 nm to 1000 nm. In this embodiment, the shape, spectral range, and resolution of the linear graded filter 4 are not limited.
[0026] Based on the minimum resolution and wavelength range of the linear graded filter 4, the effective width s of the linear graded filter 4 is divided into n filtering regions in a preset direction, and the width of each filtering region is denoted as s / n; the filtering regions are numbered sequentially from one end to the other as s1, s2...s n The center wavelengths of the transmitted light are denoted as λ1, λ2, ..., λ. n ,λ1,λ2,…,λ n The wavelengths increase sequentially; the preset direction is perpendicular to the optical axis. The center wavelength refers to the midpoint of the wavelength range of transmitted light. For example, if the wavelength range of transmitted light is 401-406, then the center wavelength is 403.
[0027] Specifically, the axial displacement stage 5 is placed parallel to the optical axis formed by the imaging lens 2 and the convex lens 3, and a linear gradient filter 4 is mounted and moved back and forth along the optical axis so that the linear gradient filter 4 coincides with the first image formed by the target object 1 after passing through the imaging lens 2, and its movement accuracy can reach 0.05 mm.
[0028] Specifically, the axial displacement stage 5 is fixed on the electrically controlled displacement stage 6. The electrically controlled displacement stage 6 drives the linear gradient filter 4 through the entire field of view along the direction perpendicular to the optical axis, i.e., the preset direction, so that the filtering area of the light transmitted by the linear gradient filter 4 with a center wavelength of λ1 moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n; the moving accuracy of the electrically controlled displacement stage 6 can reach 50 μm.
[0029] Specifically, the controller 7 is connected to the electrically controlled displacement stage 6, driving the electrically controlled displacement stage 6 to pass through the entire field of view along a direction perpendicular to the optical axis.
[0030] The imaging processing module 300 consists of an imaging camera 8 and a host computer control module 9. The imaging camera 8 is arranged coaxially at the rear end of the convex lens 3.
[0031] Specifically, the imaging camera 8 is connected to the host computer control module 9. The imaging camera acquires an image of the excised organ after each step movement of the linearly graded filter 4, obtaining an image sequence and transmitting it to the host computer control module 9. The imaging camera 8 has a spectral response range of 400 nm to 1000 nm. In this embodiment, the type of imaging camera is not limited. The controller 7 and the imaging camera are controlled by the host computer control module 9, achieving synchronous operation and hyperspectral image acquisition.
[0032] Specifically, the host computer control module 9 is independently developed based on the Python platform. Its main functions include interacting with the controller 7 to control the movement of the linear gradient filter 4, synchronously controlling the imaging camera to acquire images, reconstruct images, perform spectral correction, display images, perform single-pixel spectral reconstruction and display, and assess organ quality. In this embodiment, the development platform of the host computer control module 9 is not limited; C++, LabVIEW, MATLAB, and other development platforms can also be used to implement the host computer control module 9.
[0033] The imaging lens 2, linear gradient filter 4, convex lens 3, and imaging camera 8 are arranged in sequence along the optical axis. The focal plane of the first image obtained by the target object 1 (e.g., an excised organ) after passing through the imaging lens 2 coincides with the focal plane of the linear gradient filter 4. After passing through the convex lens 3, the first image generates an original image on the imaging camera and is transmitted to the host computer control module 9.
[0034] The host computer control module 9 is used to extract valid images p1, p2, ..., p from the image sequence. m Calculate the number of pixel columns corresponding to each filtering region of the linear gradient filter 4, denoted as t; calculate columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on until image p... mThe pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral images are obtained; the obtained hyperspectral images are classified and identified to obtain organ quality assessment results.
[0035] Specifically, the host computer control module 9 includes: an image extraction module, an image reconstruction module, and an evaluation model; The image extraction module is used to extract valid images p1, p2, ..., p from the image sequence output by the imaging camera. m Calculate the number of pixel columns corresponding to each filter region of the linear gradient filter 4, denoted as t; The image reconstruction module is used to reconstruct columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on up to image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral images; from the obtained different bands (λ1 to λ n The single-pixel spectrum of each pixel is reconstructed from the hyperspectral image. That is, the pixels at the same position in each hyperspectral image are extracted as spectral intensity information, and the wavelength corresponding to each hyperspectral image is used as wavelength information. The spectral intensity information and wavelength information constitute the single-pixel spectrum. The obtained single-pixel spectrum is then normalized and corrected to obtain the corrected single-pixel spectrum.
[0036] The evaluation model is used to classify and identify the obtained corrected single-pixel spectra to obtain organ quality assessment results.
[0037] Furthermore, the evaluation model includes: a preprocessing module, a one-dimensional convolutional neural network and a long short-term memory network, and a classification module; The preprocessing module is used to sequentially perform light source calibration, spectral dimension reconstruction and data enhancement processing on the corrected single-pixel spectrum output by the image reconstruction module to obtain the preprocessed single-pixel spectrum. The one-dimensional convolutional neural network is used to extract local features from the preprocessed single-pixel spectrum to obtain spectral data with local features. The Long Short-Term Memory (LSTM) network is used to extract global features from spectral data with local features, thereby obtaining spectral data with global features. The classification module is used to classify organ quality based on the spectral data with global features.
[0038] In this embodiment of the invention, the one-dimensional convolutional neural network includes four convolutional layers with kernel sizes of 7, 5, and 3, and channel numbers of 32, 64, 128, and 128, respectively. The second and third convolutional layers are dilated convolutional layers with dilation rates of 2 and 4, respectively, to expand the receptive field. The output of each convolutional layer is connected to a batch normalization layer and a ReLU activation function. The output of the last convolutional layer is connected to a global max pooling layer. The output of the global max pooling layer is connected to a residual connection layer. The convolutional layer is used to perform convolution operations on the preprocessed single-pixel spectrum to obtain the convolution result; The global max pooling layer is used to perform global max pooling on the convolution result to obtain the pooling result; The residual connection layer is used to perform residual connections on the pooling results to obtain spectral data with local features.
[0039] In this embodiment of the invention, the long short-term memory network includes: Spectral feature sequence recombination module: Its input is spectral data with local features extracted by a local one-dimensional convolutional neural network (shape [B,C,H,W], where B is the batch size, C is the number of channels, and H / W is the spatial dimension), and the output is a temporal feature sequence (shape [B,T,C], T=H×W). BiLSTM Temporal Modeling Module: Its input is a temporal feature sequence [B,T,C], contains 2 layers of BiLSTM, the hidden layer dimension is 64, peephole connections are introduced to enhance the temporal modeling capability, and the output is [B,T,128]. The spectral gate control module takes the current time step input x ([B,C]) and the previous hidden state h_(t-1) ([B,128]) as input and outputs the gated weighted features. (of shape [B,C]), used as input for the next time step. Wherein, , W g This is the weight matrix. b g h is the bias term. t-1 x is the output of the LSTM network at the previous time step. t This is the input to the LSTM network at the current moment; Self-attention band enhancement module: Its input is time-series feature [B,T,128], and its output is weighted time-series feature [B,T,128].
[0040] In this embodiment of the invention, the classification module includes: a global average pooling layer and a fully connected layer; A global average pooling layer is used to perform global average pooling on the spectral data with global features to obtain a feature vector of a fixed length; The fully connected layer is used to calculate the classification probability using the Softmax function based on the feature vector of the fixed length and output the organ mass evaluation result.
[0041] The host computer control module 9 includes a spectral normalization correction algorithm for performing spectral correction when acquiring single-pixel spectra to solve the problem of low quantum efficiency of the imaging camera after 700 nm. The specific algorithm steps are as follows (for reference Figure 3 ): S1: Measure the quantum efficiency curve QE(λ) of the imaging camera; S2: Extract the original single-pixel spectrum I0(λ); S3: Use the following formula to obtain the corrected single-pixel spectrum I1(λ): I1(λ) = I0(λ) / QE(λ) Pre-collect a large amount of organ hyperspectral image data for evaluation model optimization and training to improve the evaluation accuracy.
[0042] Specifically, the evaluation model optimization and training include: Adopt a two-stage training strategy, first pre-train on a large remote sensing dataset, and then fine-tune on the target dataset; use mixed-precision training to accelerate model convergence; Implement gradient clipping, set the gradient norm threshold ‖g‖ ≤ 5.0 to prevent gradient explosion during the training process; Introduce label smoothing (smoothing parameter ε = 0.1) and Dropout (dropout rate 0.3) regularization techniques to prevent overfitting.
[0043] The working method of the system of the present invention is as follows (for reference Figure 4 and Figure 5 ): S1: Drive the electric control displacement stage 6 to drive the linear variable filter 4 to move along the preset direction, with each movement distance of s / n. At the same time, for each movement of a step length, control the imaging camera to collect an image; collect an image sequence of the entire movement cycle of the filtering area s1 of the linear variable filter 4 from the start of entering the target surface of the imaging camera 8 to the complete removal from the target surface of the imaging camera 8, and record the total number of images obtained in this cycle as j; S2: The image extraction module extracts the image data of a complete cycle of the linear variable filter 4 and the target surface of the imaging camera 8 from the start of contact to complete separation from the image sequence. Let the number of effective image frames obtained in this cycle be m (where m < j, because the images collected at the front end and the back end may be invalid), and record them as p1, p2,..., p m ; S3: The image reconstruction module divides the number of pixels on the long side of the image by m to obtain the number of pixel columns corresponding to each wavelength, denoted as t; the direction of the long side of the image is consistent with the preset direction; columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on, until image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombined to obtain a new image I1, which is the hyperspectral image of the system at λ1; λ2 to λ are reconstructed sequentially using the same method. n Hyperspectral images; from the obtained different bands (λ1 to λ n The single-pixel spectrum of each pixel is reconstructed from the hyperspectral image. That is, the pixels at the same position in each hyperspectral image are extracted as spectral intensity information, and the wavelength corresponding to each hyperspectral image is used as wavelength information. The spectral intensity information and wavelength information form a single-pixel spectrum. The obtained single-pixel spectrum is then normalized and corrected to obtain the corrected single-pixel spectrum. S4: Preprocess the acquired single-pixel spectrum to obtain the preprocessed single-pixel spectrum.
[0044] Specifically, S4 mainly includes the following processing steps: S4.1: The preprocessing module performs light source calibration on each pixel of the corrected single-pixel spectrum (each image has a size of h×k) to eliminate differences in illumination intensity, resulting in h×k spectral information. The calculation formula is as follows:
[0045] in, R corrected It is the calibrated reflectance spectral intensity. R white This is the spectral image of a standard reflectivity plate. R black It is the system's dark current. R sample It is the corrected single-pixel spectrum; S4.2: Convert the calibrated h×k spectral information obtained in S4.1 into a one-dimensional vector; S4.3: Implement data augmentation strategies, including spectral jitter and spectral shift, on the one-dimensional vector obtained from S4.2 to improve the generalization ability of the model and obtain the preprocessed single-pixel spectrum.
[0046] S5: Based on a one-dimensional convolutional neural network, local features are extracted from the preprocessed single-pixel spectrum to obtain spectral data with local features.
[0047] Specifically, S5 mainly includes the following processing steps: S5.1: Construct a one-dimensional convolutional neural network containing four convolutional layers to extract local features from the preprocessed single-pixel spectrum. The kernel sizes are 7, 5, and 3, and the number of channels are 32, 64, 128, and 128, respectively. Dilated convolutions are introduced in the second and third convolutional layers with dilation rates of 2 and 4, respectively, to expand the receptive field. S5.2: Batch normalization and ReLU activation function are introduced after each convolutional layer to accelerate model convergence and enhance nonlinear expressive power; S5.3: Use global max pooling after the last convolutional layer to compress the spectral features into a fixed-length feature vector; S5.4: Design residual connections to fuse shallow features (output of the second convolutional layer) with deep features (output of the fourth convolutional layer) to alleviate the gradient vanishing problem; S5.5: Randomly select batch size h×k×n preprocessed single-pixel spectra and input them into the one-dimensional convolutional neural network constructed in S5.1-S5.4 for local feature extraction to obtain h×k spectral data with local features n1; where h and k are the pixel points of the pixel row and column of the image captured by the imaging camera, and n is the number of the filter regions.
[0048] S6: Based on the Long Short-Term Memory (LSTM) network, global feature extraction is performed on the spectral data with local features obtained in S5 to obtain spectral data with all features.
[0049] Specifically, S6 mainly includes the following processing steps: S6.1: Reassemble the spectral data with local features extracted by the one-dimensional convolutional neural network in S5 into a sequence and establish a (T×C) feature flow, where T is the number of sequence steps and C is the number of feature channels; S6.2: Construct a bidirectional LSTM network, which includes two layers of BiLSTM with a hidden layer dimension of 64, and introduce peephole connections to enhance the temporal modeling capability; S6.3: Design a spectral gating mechanism in an LSTM network. The calculation formula is as follows:
[0050] in, W g This is the weight matrix. b g h is the bias term. t-1 x is the output of the LSTM network at the previous time step. t This is the input to the LSTM network at the current moment; S6.4: Introduce a self-attention mechanism in the LSTM network to calculate the importance weights of bands and enhance the feature response of key bands.
[0051] S6.5: Input h×k spectral data with local features of n1 into the LSTM network constructed in S6 to extract global features, and obtain h×k spectral data with global features of n3.
[0052] S7: Make organ quality classification decisions based on spectral data with all characteristics.
[0053] Specifically, S7 mainly includes the following processing steps: S7.1: Compress the h×k spectral data with global features of n3 output from the LSTM network in S6 into a fixed-length feature vector through global average pooling; S7.2: Input the compressed feature vector into the fully connected layer, calculate the classification probability through the Softmax function, and output the classification result; S8: Model Training: The acquired hyperspectral data of organ quality is divided into training and validation sets in a 7:3 ratio and input into the network constructed in S5-S7 for training to obtain the optimal model parameters.
[0054] The organ quality assessment method of this invention adopts a machine learning approach that combines global and local feature extraction, and combines a one-dimensional convolutional neural network and a long short-term memory network to efficiently analyze the hyperspectral image data of organs.
Claims
1. An organ quality assessment system based on visible-near-infrared hyperspectral imaging, characterized in that, It includes an imaging lens (2), a linear gradient filter (4), a convex lens (3), and an imaging camera (8) arranged in sequence along the optical axis, as well as a drive module and a host computer control module (9). The linear gradient filter (4) is divided into n filter regions in a preset direction. The width of each filter region is denoted as s / n, and the center wavelengths of the light transmitted through each filter region from one end to the other are denoted as λ1, λ2, ..., λ. n ,λ1,λ2……λ n The wavelength range of the linear gradient filter (4) increases sequentially from 400 nm to 1000 nm; the preset direction is perpendicular to the optical axis; The driving module is used to drive the linear gradient filter (4) to move along the preset direction, so that the filter area with the center wavelength of the transmitted light being λ1 moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n. The imaging camera is used to acquire an image of the ex vivo organ after each step of the linear gradient filter (4) moves, thus obtaining an image sequence; The host computer control module (9) is used to extract valid images p1, p2, ..., p from the image sequence. m Calculate the number of pixel columns corresponding to each filtered region, denoted as t; extract columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on until image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral image; the single-pixel spectrum of each pixel is reconstructed from the hyperspectral image, and the obtained single-pixel spectra are classified and identified to obtain the organ quality assessment results.
2. The organ quality assessment system based on visible-near-infrared hyperspectral imaging according to claim 1, characterized in that, The drive module includes an axial displacement stage (5), an electrically controlled displacement stage (6), and a controller (7); a linear gradient filter (4) is disposed on the axial displacement stage (5); the axial displacement stage (5) is disposed on the electrically controlled displacement stage (6), and the axial displacement stage (5) can move along the optical axis so that the linear gradient filter (4) coincides with the focal plane of the first image formed by the isolated organ after passing through the imaging lens (2); The controller (7) is used to drive the electronically controlled displacement stage (6) to move, thereby moving the linear gradient filter (4) along the preset direction, so that the filter area with the center wavelength of the transmitted light being λ1 moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n.
3. The organ quality assessment system based on visible-near-infrared hyperspectral imaging according to claim 1, characterized in that, The imaging camera (8) has a spectral response range of 400 nm to 1000 nm.
4. The organ quality assessment system based on visible-near-infrared hyperspectral imaging according to claim 1, characterized in that, The host computer control module (9) includes: an image extraction module, an image reconstruction module, and an evaluation model; The image extraction module is used to extract valid images p1, p2, ..., p from the image sequence output by the imaging camera. m Calculate the number of pixel columns corresponding to each filter region of the linear gradient filter (4), and denote it as t; The image reconstruction module is used to reconstruct columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on up to image p... m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral image; from λ2 to λ n The hyperspectral image is used to reconstruct the single-pixel spectrum of each pixel, and the obtained single-pixel spectrum is normalized and corrected to obtain the corrected single-pixel spectrum. The evaluation model is used to classify and identify the obtained corrected single-pixel spectra to obtain organ quality assessment results.
5. The organ quality assessment system based on visible-near-infrared hyperspectral imaging according to claim 4, characterized in that, The evaluation model includes: a preprocessing module, a one-dimensional convolutional neural network and a long short-term memory network, and a classification module; The preprocessing module is used to sequentially perform light source calibration, spectral dimension reconstruction and data enhancement processing on the corrected single-pixel spectrum output by the image reconstruction module to obtain the preprocessed single-pixel spectrum. The one-dimensional convolutional neural network is used to extract local features from the preprocessed single-pixel spectrum to obtain spectral data with local features. The long short-term memory network is used to extract global features from spectral data with local features, thereby obtaining spectral data with global features. The classification module is used to classify organ quality based on the spectral data with global features.
6. The organ quality assessment system based on visible-near-infrared hyperspectral imaging according to claim 1, characterized in that, The one-dimensional convolutional neural network includes four convolutional layers, with the second and third convolutional layers being dilated convolutional layers; the output of each convolutional layer is connected to a batch normalization layer and a ReLU activation function; the output of the last convolutional layer is connected to a global max pooling layer; and the output of the global max pooling layer is connected to a residual connection layer. The convolutional layer is used to perform convolution operations on the preprocessed single-pixel spectrum to obtain the convolution result; The global max pooling layer is used to perform global max pooling on the convolution result to obtain the pooling result; The residual connection layer is used to perform residual connections on the pooling results to obtain spectral data with local features.
7. The organ quality assessment system based on visible-near-infrared hyperspectral imaging according to claim 1, characterized in that, The classification module includes: a global average pooling layer and a fully connected layer; A global average pooling layer is used to perform global average pooling on the spectral data with global features to obtain a feature vector of fixed length. The fully connected layer is used to calculate the classification probability using the Softmax function based on the fixed-length feature vector, and output the organ quality assessment result.
8. An organ quality assessment method based on visible-near-infrared hyperspectral imaging, characterized in that, The system based on any one of claims 1 to 7 includes: Place the excised organ in front of the imaging lens (2), and adjust the position of the linear gradient filter (4) so that the linear gradient filter (4) coincides with the focal plane of the first image formed by the excised organ after passing through the imaging lens (2). The linear gradient filter (4) is driven by the driving module to move along the preset direction, so that the filter area with the center wavelength of the light transmitted by the linear gradient filter (4) moves from the target surface of the imaging camera to the target surface of the imaging camera, with a step size of s / n; after each step of the linear gradient filter (4) moves, an image of the ex vivo organ is acquired by the imaging camera to obtain an image sequence. The host computer control module (9) extracts valid images p1, p2, ..., p from the image sequence. m Calculate the number of pixel columns corresponding to each filtering region of the linear gradient filter (4), denoted as t; calculate columns 1 to t of image p1, columns t+1 to 2t of image p2, and so on until image p m The pixels from column (m-1)×t+1 to column m×t are sequentially recombine to obtain the hyperspectral image corresponding to λ1; and so on, to obtain λ2 to λ... n The corresponding hyperspectral image is used to reconstruct the single-pixel spectrum of each pixel; the obtained single-pixel spectra are then classified and identified to obtain the organ quality assessment results.
9. The organ quality assessment method based on visible-near-infrared hyperspectral imaging according to claim 8, characterized in that, The process of classifying and identifying the obtained single-pixel spectra to obtain organ quality assessment results includes: The single-pixel spectrum is normalized, calibrated, reconstructed in spectral dimensions, and augmented with data to obtain the preprocessed single-pixel spectrum. Local features are extracted from the preprocessed single-pixel spectrum to obtain spectral data with local features; Global feature extraction is performed on spectral data with local features to obtain spectral data with global features; Based on the spectral data with global characteristics, organ quality is classified.