Near-infrared fluorescence imaging method based on Gram matrix and style domain transfer

Abstract

The present invention belongs to the field of biomedical molecular imaging, and specifically relates to a near-infrared fluorescence imaging method, system, and device based on Gram matrix and style domain conversion, which aims to address the shortcomings of nanoparticle probes in the long-wavelength window of the near-infrared region II, such as toxicity and lack of regulatory approval. The present invention comprises the following steps: obtaining high-quality fluorescence images and low-quality fluorescence images and using them as training sets, constructing and training a generative adversarial network based on Gram matrix and style domain conversion, and inputting the real acquired data into the trained network to obtain high-quality fluorescence images and cyclically transformed fluorescence images in the style of the long-wavelength near-infrared region II window. The machine learning-based method of the present invention can improve the accuracy of near-infrared fluorescence imaging technology, which has been widely used in clinical practice, without having to worry about differences in specific optical parameters and biological characteristics; it is conducive to studying the specific distribution of fluorescent probes in the body, and is of great significance to clinical decision-making and practice.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical molecular imaging, and specifically relates to a near-infrared fluorescence imaging method, system and device based on Gram matrix and style domain conversion. Background Art

[0002] Near-infrared fluorescence imaging has become an emerging in vivo imaging method with high spatiotemporal resolution at millimeter tissue depths. Imaging in the long-wavelength near-infrared zone II window (>1500nm) can effectively suppress light scattering and maximize imaging penetration depth, with high-quality imaging characteristics. However, nanoparticle probes for imaging in the long-wavelength near-infrared zone II window have disadvantages such as toxicity and lack of regulatory approval, making clinical translation difficult. On the other hand, biosafe fluorescent probes can be used for imaging in the short-wavelength near-infrared window (700-1300nm), but when imaging in this window, light scattering and autofluorescence seriously affect the tissue-to-background signal ratio and its resolution. Therefore, new near-infrared fluorescence imaging methods are urgently needed that can improve the tissue-to-background signal ratio and resolution during imaging, and perform batch, reference-free quantitative analysis of images, so as to achieve high-quality imaging under the conditions of using biosafe fluorescent probes.

[0003] The use of near-infrared fluorescence imaging methods and devices based on Gram matrix and style domain conversion can improve the imaging quality of near-infrared images, thereby more accurately depicting complex tissue morphology and determining tumor boundaries, and further expanding the application of fluorescence imaging in pre-clinical and clinical aspects such as vascular imaging, guided surgery, tumor diagnosis and treatment, and pharmacokinetics.

[0004] Based on this, the present invention provides a near-infrared fluorescence imaging method based on Gram matrix and style domain conversion. Summary of the Invention

[0005] In order to solve the above-mentioned problems in the prior art, namely, the problems that nanoparticle probes for imaging in the long wavelength window of the near-infrared region II have disadvantages such as toxicity and lack of regulatory approval, making them difficult to clinically transform, the present invention provides a near-infrared fluorescence imaging method based on Gram matrix and style domain conversion, the method comprising:

[0006] Low-quality near-infrared fluorescence images of the organism to be imaged are collected and fed into a trained generative adversarial network based on the Gram matrix and style domain conversion to obtain high-quality fluorescence images with a long-wavelength near-infrared second-zone window style and cyclically transformed fluorescence images.

[0007] The training method of the generative adversarial network based on Gram matrix and style domain conversion is as follows:

[0008] Step S100, acquiring a high-quality fluorescence image and a low-quality fluorescence image;

[0009] The high-quality fluorescence image is a near-infrared fluorescence image generated in the long-wavelength near-infrared second zone window after the emission of the fluorophore and multi-parameter transformation integration; the low-quality fluorescence image is a short-wavelength near-infrared window fluorescence image emitted by the fluorophore;

[0010] Step S200: Using the high-quality fluorescence image and the low-quality fluorescence image as training sets, constructing and training a generative adversarial network based on Gram matrix and style domain conversion.

[0011] In some preferred embodiments, the high-quality fluorescence image is obtained by:

[0012] Step S110: collecting unmatched short-wavelength near-infrared window fluorescence images and unpaired long-wavelength near-infrared second-zone window fluorescence images of the same image size, aggregating the long-wavelength near-infrared second-zone window fluorescence images, integrating them into a target conversion domain of a near-infrared fluorescence image, and obtaining an integrated fluorescence image;

[0013] Step S120: performing a multi-parameter geometric transformation on the integrated fluorescence image to obtain an enhanced high-quality fluorescence image;

[0014] The multi-parameter geometric transformation includes one or more of rotation transformation, translation transformation, Euclidean transformation, and affine transformation.

[0015] In some preferred embodiments, the generative adversarial network based on the Gram matrix and style domain conversion is constructed by:

[0016] A generative adversarial network based on Gram matrix and style domain conversion is constructed through a first generator and a second generator; the first generator is a generator of domain style encoding and decoding through image forward conversion; the second generator is a generator of domain style encoding and decoding through image inverse conversion;

[0017] The first generator and the second generator are both composed of a domain style encoding network and a domain style decoding network.

[0018] In some preferred embodiments, the training method of the generative adversarial network based on the Gram matrix and style domain conversion is:

[0019] Step S210: inputting the low-quality fluorescence image into the domain style encoding network in the first generator to obtain a first encoding feature map;

[0020] Step S220: input the first encoded feature map into the domain style decoding network in the first generator to obtain a first training image;

[0021] Step S230: input the first training image into the domain style encoding network in the second generator to obtain a second encoding feature map;

[0022] Step S240: input the second encoded feature map into the domain style decoding network in the second generator to obtain a second training image;

[0023] Step S250: inputting the high-quality fluorescence image into the domain style encoding network in the second generator to obtain a third encoding feature map;

[0024] Step S260: input the third encoded feature map into the domain style decoding network in the second generator to obtain a third training image;

[0025] Step S270: input the third training image into the domain style encoding network in the first generator to obtain a fourth encoding feature map;

[0026] Step S280: Input the fourth encoded feature map into the domain style decoding network in the first generator to obtain a fourth training image.

[0027] In some preferred embodiments, the training method of the generative adversarial network based on the Gram matrix and style domain conversion further includes:

[0028] Step S290: Calculate the style correlation between the first training image and the high-quality fluorescent image, the style correlation between the second training image and the low-quality fluorescent image, the style correlation between the third training image and the low-quality fluorescent image, and the style correlation between the fourth training image and the high-quality fluorescent image using a Gramian matrix, and modify the generative adversarial network based on the Gramian matrix and style domain conversion according to the style correlations.

[0029] Step S300: Calculate the semantic relevance between the second training image and the low-quality fluorescence image, and the semantic relevance between the fourth training image and the high-quality fluorescence image, and modify the generative adversarial network based on the Gram matrix and style domain conversion according to the semantic relevance.

[0030] In some preferred implementations, the style correlation is calculated as follows:

[0031] Get the inner product between the vectorized feature map a and the vectorized feature map b at position i in the cth layer of the convolutional neural network

[0032]

[0033] Among them, the H c is the vectorized feature map matrix of the c-th layer of the image;

[0034] Calculate the weighted correlation of N-layer style features. When x is a high-quality fluorescence image and y is the first training image and the fourth training image, represent the smooth multi-scale representation between the high-quality fluorescence image and the first training image and the fourth training image, and capture the style information to obtain the style correlation L s ;

[0035] When x is the low-quality fluorescence image and y is the second training image and the third training image, a smooth multi-scale representation between the low-quality fluorescence image and the second training image and the third training image is represented, and style information is captured to obtain the style correlation L s :

[0036]

[0037] Among them, w c is the weight value of the c-th layer, and the vectorized feature map a and the vectorized feature map b are two different feature maps of x at the c-th layer.

[0038] In some preferred embodiments, the method for calculating semantic relevance is:

[0039] Obtain the deep features F of the convolutional neural network to represent the semantic content of the image, calculate the weighted correlation of the M-layer semantic features, obtain the high-level semantic representation between the high-quality fluorescent image m and the fourth training image n, and the high-level semantic representation between the low-quality fluorescent image m and the second training image n, as the semantic correlation L between the fourth training image and the high-quality fluorescent image, and the second training image and the low-quality fluorescent image co :

[0040]

[0041] where w d is the weight value of the dth layer.

[0042] In some preferred embodiments, a pair of conjugate mappings from the low-quality fluorescence image to the high-quality fluorescence image and from the high-quality fluorescence image to the low-quality fluorescence image are learned by a cyclic training method.

[0043] In some preferred embodiments, a low-quality near-infrared fluorescence image corresponding to an organism to be subjected to near-infrared fluorescence imaging is collected and input into a trained generative adversarial network based on a Gram matrix and style domain conversion to obtain a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style and a cyclically transformed fluorescence image, the method being:

[0044] After injecting a fluorescent probe into a living organism, a low-quality near-infrared fluorescence image of the living organism is obtained, and the low-quality near-infrared fluorescence image is input into the trained generative adversarial network based on the Gram matrix and style domain conversion, and a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style is obtained through a domain style encoding and decoding network of image positive conversion;

[0045] The high-quality fluorescence image is input into the trained generative adversarial network based on Gram matrix and style domain conversion, and a cyclically transformed fluorescence image is obtained through the domain style encoding and decoding network of image inverse conversion.

[0046] In some preferred embodiments, after obtaining the high-quality fluorescence image with long-wavelength near-infrared second-region window style and the cyclically transformed fluorescence image output by the trained generative adversarial network based on the Gram matrix and style domain conversion, the method further includes:

[0047] The high-quality fluorescence image with long-wavelength near-infrared second-zone window style and the cyclic transformation fluorescence image are subjected to a no-truth evaluation method, and quantitative analysis results are obtained under the condition of limited paired acquisition:

[0048] Inputting the output cyclically transformed fluorescence image and the low-quality fluorescence image into a structural similarity evaluator to obtain a quantized structural similarity measurement result; inputting the output cyclically transformed fluorescence image and the low-quality fluorescence image into a root mean square error evaluator to obtain a quantized root mean square error measurement result;

[0049] The output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into the natural image quality evaluator to obtain a quantitative natural image quality evaluation result; the output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into the visual perception evaluator to obtain a quantitative visual quality evaluation result.

[0050] Beneficial effects of the present invention:

[0051] (1) The generative adversarial network based on Gram matrix and style domain conversion trained by the present invention can convert the fluorescence image in the short-wavelength near-infrared window into a high-quality fluorescence image with the style of the long-wavelength near-infrared second zone window;

[0052] (2) The machine learning-based method of the present invention can improve the accuracy of near-infrared fluorescence imaging technology, which has been widely used in clinical practice, without having to consider the differences in specific optical parameters and biological characteristics;

[0053] (3) The present invention can be widely used in vascular imaging, neuroimaging, and determining tumor boundaries, and is conducive to studying the specific distribution of fluorescent probes in the body, which is of great significance to clinical decision-making and practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0055] Figure 1 It is a flow chart of the near-infrared fluorescence imaging method based on Gram matrix and style domain conversion of the present invention;

[0056] Figure 2 1 is a structural diagram of a generative adversarial network based on Gram matrix and style domain conversion according to the present invention;

[0057] Figure 3 1 is a diagram showing the structure of a generator of a generative adversarial network based on Gram matrix and style domain conversion according to the present invention;

[0058] Figure 4 This is a comparison chart of the results of ordinary near-infrared fluorescence imaging and the near-infrared fluorescence imaging method based on Gram matrix and style domain conversion;

[0059] Figure 5 It is a schematic diagram of the structure of a computer system of a server for implementing the embodiments of the method, system, and apparatus of the present application;

[0060] Table 1 is a schematic diagram of the visual quality evaluation results and the root mean square error measurement results.

[0061] DETAILED DESCRIPTION

[0062] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0063] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] See also Figure 1-4 The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to the first embodiment of the present invention comprises the following steps:

[0065] Low-quality near-infrared fluorescence images of the organism to be imaged are collected and fed into a trained generative adversarial network based on the Gram matrix and style domain conversion to obtain high-quality fluorescence images with a long-wavelength near-infrared second-zone window style and cyclically transformed fluorescence images.

[0066] The training method of the generative adversarial network based on Gram matrix and style domain conversion is as follows:

[0067] Step S100, acquiring a high-quality fluorescence image and a low-quality fluorescence image;

[0068] The high-quality fluorescence image is a near-infrared fluorescence image generated in the long-wavelength near-infrared second zone window after the emission of the fluorophore and multi-parameter transformation integration; the low-quality fluorescence image is a short-wavelength near-infrared window fluorescence image emitted by the fluorophore;

[0069] Step S200: Using the high-quality fluorescence image and the low-quality fluorescence image as training sets, constructing and training a generative adversarial network based on Gram matrix and style domain conversion.

[0070] Preferably, the high-quality fluorescence image is obtained by:

[0071] Step S110: collecting unmatched short-wavelength near-infrared window fluorescence images and unpaired long-wavelength near-infrared second-zone window fluorescence images of the same image size, aggregating the long-wavelength near-infrared second-zone window fluorescence images, integrating them into a target conversion domain of a near-infrared fluorescence image, and obtaining an integrated fluorescence image;

[0072] Step S120: performing a multi-parameter geometric transformation on the integrated fluorescence image to obtain an enhanced high-quality fluorescence image;

[0073] The multi-parameter geometric transformation includes one or more of rotation transformation, translation transformation, Euclidean transformation, and affine transformation;

[0074] Step S130: All final images were randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1 for model training. The training process was performed using a computing platform equipped with an NVIDIA GPU (Tesla V100 with 32GB). The learning rate of the neural network was set to 0.0002, and the optimization algorithm used was Adam, with β1 = 0.9 and β2 = 0.99.

[0075] Preferably, see Figure 2 , the generative adversarial network based on Gram matrix and style domain conversion is constructed as follows:

[0076] A generative adversarial network based on Gram matrix and style domain conversion is constructed through a first generator and a second generator; the first generator is a generator of domain style encoding and decoding through image forward conversion; the second generator is a generator of domain style encoding and decoding through image inverse conversion;

[0077] See also Figure 3, the first generator and the second generator are both composed of a domain style encoding network and a domain style decoding network;

[0078] The domain style encoding networks in the first generator and the second generator are both composed of alternating combinations and jump connections of double convolutional downsampling units and maximum pooling layers;

[0079] Among them, the specific method of the alternating combination is: the feature map is first input into the double convolution downsampling unit and then passes through the maximum pooling layer, and then the new feature map is output and the above steps are repeated until the set number of alternating combination layers is reached; the specific method of the jump connection is: the input of each double convolution unit is element-by-element added to the output of the double convolution unit through the jump connection, and then input into the maximum pooling layer.

[0080] The style decoding networks in the first generator and the second generator are both composed of upsampling layers and double convolutional downsampling units alternately combined and skipped;

[0081] The domain style encoding network and the domain style decoding network in the first generator and the second generator are connected via dual convolutional downsampling units. The dual convolutional downsampling units that output feature maps of the same size transmit feature information streams via skip connections, and the feature information maps are element-wise summed.

[0082] The dual convolution downsampling unit includes a convolution downsampling layer, an instance normalization layer, an activation function layer and a neuron random inactivation function that are weightedly connected to each other, and uses identity mapping for feature merging and activation function for nonlinear activation;

[0083] Both local mean-based discriminators are composed of local convolutional neural networks, which are used to divide the image and perform targeted discrimination in different areas, which can help the generator to perform targeted enhancement of different tissues and details.

[0084] Among them, the neuron random inactivation function is used to inactivate local neurons to prevent overfitting of the neural network; the encoded feature map input double convolution downsampling unit passes through the convolution downsampling layer, instance normalization layer, activation function layer, and neuron random inactivation function respectively; weighted connections are used between each layer of the network, and the ReLU activation function is used for nonlinear activation. The formula is as follows:

[0085]

[0086] in represents the i-th neuron in the n-th layer network, represents the jth neuron in the n+1th layer network, w i Represents the connection weight of the i-th neuron, ReLU represents the activation function, and the formula is as follows:

[0087]

[0088] After the image is input into the discriminator, it passes through the convolutional downsampling layer, the activation function layer, and the batch normalization layer to obtain a batch feature map. Local mean merging is performed to output a discriminant label for the image's authenticity. The dual convolutional downsampling unit includes a convolutional layer, an instance normalization layer, and an activation function layer. Weighted connections are used between each network layer, and the LeakyReLU activation function is used for nonlinear activation. The formula is as follows:

[0089]

[0090] The value range of the a parameter is (1, +∞).

[0091] Preferably, see Figure 2 , the training method of the generative adversarial network based on Gram matrix and style domain conversion is:

[0092] Step S210: inputting the low-quality fluorescence image into the domain style encoding network in the first generator to obtain a first encoding feature map;

[0093] Step S220: input the first encoded feature map into the domain style decoding network in the first generator to obtain a first training image;

[0094] Step S230: input the first training image into the domain style encoding network in the second generator to obtain a second encoding feature map;

[0095] Step S240: input the second encoded feature map into the domain style decoding network in the second generator to obtain a second training image;

[0096] Step S250: inputting the high-quality fluorescence image into the domain style encoding network in the second generator to obtain a third encoding feature map;

[0097] Step S260: input the third encoded feature map into the domain style decoding network in the second generator to obtain a third training image;

[0098] Step S270: input the third training image into the domain style encoding network in the first generator to obtain a fourth encoding feature map;

[0099] Step S280: Input the fourth encoded feature map into the domain style decoding network in the first generator to obtain a fourth training image.

[0100] Preferably, see Figure 2The training method of the generative adversarial network based on Gram matrix and style domain conversion further includes:

[0101] Step S290: Calculate the style correlation between the first training image and the high-quality fluorescent image, the style correlation between the second training image and the low-quality fluorescent image, the style correlation between the third training image and the low-quality fluorescent image, and the style correlation between the fourth training image and the high-quality fluorescent image using a Gramian matrix, and modify the generative adversarial network based on the Gramian matrix and style domain conversion according to the style correlations.

[0102] Step S300: Calculate the semantic relevance between the second training image and the low-quality fluorescence image, and the semantic relevance between the fourth training image and the high-quality fluorescence image, and modify the generative adversarial network based on the Gram matrix and style domain conversion according to the semantic relevance.

[0103] Preferably, see Figure 2 , the method for calculating the style correlation is:

[0104] Get the inner product between the vectorized feature map a and the vectorized feature map b at position i in the cth layer of the convolutional neural network

[0105]

[0106] The H c is the vectorized feature map matrix of the cth layer of the image; wherein, c is any layer of the network in this network.

[0107] The Gram matrix style representation is built on the convolutional layer of the convolutional neural network for semantic style extraction and calculates the style representation between the output features of the shallow convolution kernel. The weighted correlation of the N-layer style features is calculated. When x is a high-quality fluorescent image and y is the first training image and the fourth training image, it represents the smooth multi-scale representation between the high-quality fluorescent image and the first training image and the fourth training image, and captures the style information to obtain the style correlation L. s ;

[0108] When x is the low-quality fluorescence image and y is the second training image and the third training image, a smooth multi-scale representation between the low-quality fluorescence image and the second training image and the third training image is represented, and style information is captured to obtain the style correlation L s :

[0109]

[0110] Among them, wc is the weight value of the c-th layer, the vectorized feature map a and the vectorized feature map b are two different feature maps of x at the c-th layer, and Ls represents the style correlation error.

[0111] The Gram matrix style representation cycle loss is introduced to constrain the style of the same cycle transformation domain. The Gram matrix style representation loss is introduced to constrain the style of the same transformation domain for the training of generative adversarial networks based on Gram matrix and style domain transformation.

[0112] Among them, the Gram matrix style representation constraint can achieve the consistency of cyclic conversion style and target domain conversion style in the feature space, thereby reducing imaging artifacts.

[0113] Preferably, see Figure 2 , the method for calculating the semantic relevance is:

[0114] Obtain the deep features F of the convolutional neural network to represent the semantic content of the image, calculate the weighted correlation of the M-layer semantic features, obtain the high-level semantic representation between the high-quality fluorescent image m and the fourth training image n, and the high-level semantic representation between the low-quality fluorescent image m and the second training image n, as the semantic correlation L between the fourth training image and the high-quality fluorescent image, and the second training image and the low-quality fluorescent image co :

[0115]

[0116] where w d is the weight value of the dth layer, L co stands for semantic relevance error.

[0117] Semantic content loss is introduced to constrain single-shot conversion for training generative adversarial networks based on Gram matrix and style domain conversion.

[0118] Preferably, a pair of conjugate mappings from the low-quality fluorescent image to the high-quality fluorescent image and from the high-quality fluorescent image to the low-quality fluorescent image are learned through a cyclic training method, and a cyclic consistency loss is introduced to constrain the domain conversion for training a generative adversarial network based on the Gram matrix and style domain conversion.

[0119] Preferably, see Figure 4 , collect low-quality near-infrared fluorescence images corresponding to the organism to be imaged, and input them into the trained generative adversarial network based on Gram matrix and style domain conversion to obtain high-quality fluorescence images with long-wavelength near-infrared second-zone window style and cyclic transformation fluorescence images. The method is as follows:

[0120] After injecting a fluorescent probe into a living organism, a low-quality near-infrared fluorescence image of the living organism is obtained, and the low-quality near-infrared fluorescence image is input into the trained generative adversarial network based on the Gram matrix and style domain conversion, and a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style is obtained through a domain style encoding and decoding network of image positive conversion;

[0121] The high-quality fluorescence image is input into the trained generative adversarial network based on Gram matrix and style domain conversion, and a cyclically transformed fluorescence image is obtained through the domain style encoding and decoding network of image inverse conversion.

[0122] In this example, the effectiveness was verified in the fluorescence images of mice. The probe was injected into the mouse through the tail vein and the mouse blood vessels were captured in the short wavelength near infrared window. Figure 4 As shown in (a), the short-wavelength near-infrared window image is input into the trained neural network, and the domain style encoding and decoding network of the image positive conversion is used to obtain a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style. Figure 4 As shown in (b);

[0123] The output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into a trained generative adversarial network based on Gram matrix and style domain conversion, and a cyclically transformed fluorescence image is obtained through the domain style encoding and decoding network of image inverse conversion.

[0124] The output high-quality image is compared with the original image and quantitative analysis is performed to obtain the quantitative index improvement results.

[0125] Preferably, referring to Table 1, after obtaining the high-quality fluorescence image with long-wavelength near-infrared second-region window style and the cyclically transformed fluorescence image output by the trained generative adversarial network based on the Gram matrix and style domain conversion, the method further includes:

[0126] The high-quality fluorescence image with long-wavelength near-infrared second-zone window style and the cyclic transformation fluorescence image are subjected to a no-truth value evaluation method, and quantitative analysis results are obtained under the condition of limited paired acquisition, which include:

[0127] Inputting the output cyclically transformed fluorescence image and the low-quality fluorescence image into a structural similarity evaluator to obtain a quantized structural similarity measurement result; inputting the output cyclically transformed fluorescence image and the low-quality fluorescence image into a root mean square error evaluator to obtain a quantized root mean square error measurement result to present the fidelity of the proposed method;

[0128] The output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into the natural image quality evaluator to obtain the quantitative natural image quality evaluation result; the output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into the visual perception evaluator to obtain the quantitative visual quality evaluation result to show the visual quality improvement of the proposed method.

[0129] Table 1 shows the degree of improvement in fidelity and visual performance of this embodiment compared to the traditional near-infrared two-zone imaging method. Among them, the structural similarity is positively correlated with the fidelity of the image, and the structural similarity measures the high fidelity between the imaged picture and the picture obtained by the traditional imaging method (the index range is 0.0000-1.0000). The root mean square error is negatively correlated with the fidelity of the image, and the root mean square error measures the pixel-level difference between the imaged picture and the picture obtained by the traditional imaging method. The fidelity is high (the index range is 0.0000-1.0000). The natural image quality evaluation index is negatively correlated with the visual quality of the image, and the natural image quality of the imaged picture is improved by 10.98% compared to the picture obtained by the traditional imaging method. The visual perception quality evaluation index is negatively correlated with the visual quality of the image, and the visual perception quality of the imaged picture is improved by 20.83% compared to the picture obtained by the traditional imaging method.

[0130] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0131] A near-infrared fluorescence imaging system based on Gram matrix and style domain conversion according to a second embodiment of the present invention is provided. The system is based on a near-infrared fluorescence imaging method based on Gram matrix and style domain conversion. The system includes: an acquisition module and a training module;

[0132] The acquisition module is configured to acquire a low-quality near-infrared fluorescence image corresponding to the organism to be subjected to near-infrared fluorescence imaging, and input the image into a trained generative adversarial network based on the Gram matrix and style domain conversion to obtain a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style and a cyclically transformed fluorescence image;

[0133] The training method of the generative adversarial network based on Gram matrix and style domain conversion is as follows:

[0134] The training module is configured to acquire high-quality fluorescence images and low-quality fluorescence images;

[0135] The high-quality fluorescence image is a near-infrared fluorescence image generated by the fluorophore in the long-wavelength near-infrared second zone window, and is obtained by multi-parameter transformation and integration; the low-quality fluorescence image is a short-wavelength near-infrared window fluorescence image emitted by the fluorophore;

[0136] The high-quality fluorescence images and the low-quality fluorescence images are used as training sets to construct and train a generative adversarial network based on Gram matrix and style domain conversion.

[0137] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0138] It should be noted that the near-infrared fluorescence imaging system based on Gram matrix and style domain conversion provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into a single module or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are merely for the purpose of distinguishing the modules or steps and are not to be considered as improper limitations of the present invention.

[0139] An electronic device according to a third embodiment of the present invention includes:

[0140] At least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-mentioned near-infrared fluorescence imaging method based on Gram matrix and style domain conversion.

[0141] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned near-infrared fluorescence imaging method based on Gram matrix and style domain conversion.

[0142] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0144] Reference below Figure 5 , which shows a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 5 The server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0145] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 into the random access memory (RAM) 503. Various programs and data required for system operation are also stored in the RAM 503. The CPU 501, ROM 502 and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0146] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 510 as needed so that a computer program read therefrom can be installed into the storage section 508 as needed.

[0147] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0148] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0149] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0150] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0151] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0152] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A near-infrared fluorescence imaging method based on Gram matrix and style domain conversion, characterized in that: The method comprises: Low-quality near-infrared fluorescence images of the organism to be imaged are collected and fed into a trained generative adversarial network based on the Gram matrix and style domain conversion to obtain high-quality fluorescence images with a long-wavelength near-infrared second-zone window style and cyclically transformed fluorescence images. The training method of the generative adversarial network based on Gram matrix and style domain conversion is as follows: Step S100, acquiring a high-quality fluorescence image and a low-quality fluorescence image; The high-quality fluorescence image is a near-infrared fluorescence image generated in the long-wavelength near-infrared second zone window after the emission of the fluorophore and multi-parameter transformation integration; the low-quality fluorescence image is a short-wavelength near-infrared window fluorescence image emitted by the fluorophore; Step S200: Using the high-quality fluorescence image and the low-quality fluorescence image as training sets, constructing and training a generative adversarial network based on a Gram matrix and style domain conversion; The generative adversarial network based on Gram matrix and style domain conversion is constructed as follows: A generative adversarial network based on Gram matrix and style domain conversion is constructed through a first generator and a second generator; the first generator is a generator of domain style encoding and decoding through image forward conversion; the second generator is a generator of domain style encoding and decoding through image inverse conversion; The first generator and the second generator are both composed of a domain style encoding network and a domain style decoding network; The training method of the generative adversarial network based on the Gram matrix and style domain conversion includes: Processing the low-quality fluorescence image through a domain style encoding network and a domain style decoding network of the first generator to obtain a first training image; Processing the first training image sequentially through the domain style encoding network and the domain style decoding network of the second generator to obtain a second training image; Processing the high-quality fluorescence image sequentially through the domain style encoding network and the domain style decoding network of the second generator to obtain a third training image; Processing the third training image sequentially through the domain style encoding network and the domain style decoding network of the first generator to obtain a fourth training image; calculating, by using a Gram matrix, style correlations between the first training image and the high-quality fluorescence image, between the second training image and the low-quality fluorescence image, between the third training image and the low-quality fluorescence image, and between the fourth training image and the high-quality fluorescence image; The semantic correlation between the second training image and the low-quality fluorescent image, and the semantic correlation between the fourth training image and the high-quality fluorescent image are calculated, and the generative adversarial network based on the Gram matrix and style domain conversion is modified according to the semantic correlation and style correlation.

2. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: The high-quality fluorescence image is obtained by: Step S110: collecting unmatched short-wavelength near-infrared window fluorescence images and unpaired long-wavelength near-infrared second-zone window fluorescence images of the same image size, aggregating the long-wavelength near-infrared second-zone window fluorescence images, integrating them into a target conversion domain of a near-infrared fluorescence image, and obtaining an integrated fluorescence image; Step S120: performing a multi-parameter geometric transformation on the integrated fluorescence image to obtain an enhanced high-quality fluorescence image; The multi-parameter geometric transformation includes one or more of rotation transformation, translation transformation, Euclidean transformation, and affine transformation.

3. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: The training method of the generative adversarial network based on Gram matrix and style domain conversion is as follows: Step S210: inputting the low-quality fluorescence image into the domain style encoding network in the first generator to obtain a first encoding feature map; Step S220: input the first encoded feature map into the domain style decoding network in the first generator to obtain a first training image; Step S230: input the first training image into the domain style encoding network in the second generator to obtain a second encoding feature map; Step S240: input the second encoded feature map into the domain style decoding network in the second generator to obtain a second training image; Step S250: inputting the high-quality fluorescence image into the domain style encoding network in the second generator to obtain a third encoding feature map; Step S260: input the third encoded feature map into the domain style decoding network in the second generator to obtain a third training image; Step S270: input the third training image into the domain style encoding network in the first generator to obtain a fourth encoding feature map; Step S280: Input the fourth encoded feature map into the domain style decoding network in the first generator to obtain a fourth training image.

4. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: The style correlation method is calculated as follows: Get the inner product between the vectorized feature map a and the vectorized feature map b at position i in the cth layer of the convolutional neural network : ； Among them, the is the vectorized feature map matrix of the c-th layer of the image; Calculate the weighted correlation of N layers of style features. When x is a high-quality fluorescent image and y is the first training image and the fourth training image, represent the smooth multi-scale representation between the high-quality fluorescent image and the first training image and the fourth training image, and capture the style information to obtain the style correlation. ; When x is the low-quality fluorescence image and y is the second training image and the third training image, a smooth multi-scale representation between the low-quality fluorescence image and the second training image and the third training image is represented, and style information is captured to obtain style correlation. : ； Among them, w c is the weight value of the c-th layer, and the vectorized feature map a and the vectorized feature map b are two different feature maps of x at the c-th layer.

5. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: The method for calculating the semantic relevance is: Obtain the deep features F of the convolutional neural network to represent the semantic content of the image, calculate the weighted correlation of the semantic features of the M layers, obtain the high-level semantic representation between the high-quality fluorescent image m and the fourth training image n, and the high-level semantic representation between the low-quality fluorescent image m and the second training image n, as the semantic correlation between the fourth training image and the high-quality fluorescent image, and the second training image and the low-quality fluorescent image : ； where w d is the weight value of the dth layer.

6. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: A pair of conjugate mappings from the low-quality fluorescence image to the high-quality fluorescence image and from the high-quality fluorescence image to the low-quality fluorescence image are learned by a cyclic training method.

7. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: A low-quality near-infrared fluorescence image corresponding to the organism to be imaged is collected and input into a trained generative adversarial network based on the Gram matrix and style domain conversion to obtain a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style and a cyclic transformation fluorescence image. The method is as follows: After injecting a fluorescent probe into a living organism, a low-quality near-infrared fluorescence image of the living organism is obtained, and the low-quality near-infrared fluorescence image is input into the trained generative adversarial network based on the Gram matrix and style domain conversion, and a high-quality fluorescence image with a long-wavelength near-infrared second-zone window style is obtained through a domain style encoding and decoding network of image positive conversion; The high-quality fluorescence image is input into the trained generative adversarial network based on Gram matrix and style domain conversion, and a cyclically transformed fluorescence image is obtained through the domain style encoding and decoding network of image inverse conversion.

8. The near-infrared fluorescence imaging method based on Gram matrix and style domain conversion according to claim 1, characterized in that: After obtaining the high-quality fluorescence image with long-wavelength near-infrared second-region window style and the cyclically transformed fluorescence image output by the trained generative adversarial network based on the Gram matrix and style domain conversion, the method further includes: The high-quality fluorescence image with long-wavelength near-infrared second-zone window style and the cyclic transformation fluorescence image are subjected to a no-truth evaluation method, and quantitative analysis results are obtained under the condition of limited paired acquisition: Inputting the output cyclically transformed fluorescence image and the low-quality fluorescence image into a structural similarity evaluator to obtain a quantized structural similarity measurement result; inputting the output cyclically transformed fluorescence image and the low-quality fluorescence image into a root mean square error evaluator to obtain a quantized root mean square error measurement result; The output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into the natural image quality evaluator to obtain a quantitative natural image quality evaluation result; the output high-quality fluorescence image with long-wavelength near-infrared second-zone window style is input into the visual perception evaluator to obtain a quantitative visual quality evaluation result.

Images