Fluorescence imaging method and fluorescence imaging apparatus

Abstract

The present disclosure provides a fluorescence imaging method and a fluorescence imaging device, the method comprising: acquiring a signal accumulation fluorescence image; inputting the signal accumulation fluorescence image into a trained fluorescence imaging model to output a denoised target fluorescence image, wherein the fluorescence imaging model is constructed according to a gradient-guided descent model and an adaptive gated denoising network, and the fluorescence imaging method of the present disclosure effectively improves the imaging accuracy of near-infrared fluorescence imaging technology and can be widely applied in the fields of blood vessel imaging, neural imaging, tumor boundary imaging, etc.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a fluorescence imaging method, a fluorescence imaging device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Near-infrared fluorescence imaging is an emerging imaging method with good molecular specificity and high spatiotemporal resolution at millimeter tissue depths. Poor signal-to-noise ratio due to weak photon signals severely impacts image quality. On one hand, the low photon yield and low concentration of fluorescent probes result in weak photon signals. Although higher excitation power can be used to increase fluorescent photons, living cells and organisms are fragile and cannot withstand high excitation doses. On the other hand, noise-reduced images can be obtained through slow, high-signal accumulation without damaging the living system; however, in vivo imaging requires high imaging speeds for observing dynamic events. Therefore, there is an urgent need for new near-infrared fluorescence imaging methods that can improve the signal-to-noise ratio during imaging, thereby achieving rapid and high-quality in vivo imaging.

[0003] Near-infrared fluorescence imaging methods and devices based on proximal gradient descent and adaptive gating noise reduction networks can improve the imaging quality of near-infrared images, thereby more accurately depicting the aggregation of weak signal probes in tissues. This can be used for tumor boundary localization and further expand the application of fluorescence imaging in preclinical and clinical fields such as vascular imaging, guided surgery, tumor diagnosis and treatment, and pharmacokinetics. Summary of the Invention

[0004] In view of the above, embodiments of this disclosure provide a fluorescence imaging method, a fluorescence imaging device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] One aspect of this disclosure provides a fluorescence imaging method, including:

[0006] Acquire fluorescence images of signal accumulation;

[0007] The accumulated fluorescence image is input into a trained fluorescence imaging model, which outputs a denoised target fluorescence image. The fluorescence imaging model is constructed based on a differentiable gradient descent model and an adaptive gated denoising network.

[0008] According to embodiments of this disclosure, the fluorescence imaging model described above is trained in the following manner:

[0009] Obtain a fluorescence image training set, wherein the fluorescence image training set includes multiple initial signal accumulation training images and image labels corresponding to each of the initial signal accumulation training images;

[0010] The differentiable gradient descent model described above is used to process multiple initial signal accumulation training images to obtain gradient variable parameters corresponding to each initial signal accumulation training image.

[0011] The adaptive gated denoising network described above is used to process multiple gradient variable parameters to obtain an intermediate output image corresponding to each of the initial signal accumulation training images. The initial image processing model includes the differentiable gradient descent model and the adaptive gated denoising network described above.

[0012] The intermediate output image is used as a new initial signal to accumulate training images for iterative processing using the initial image processing model.

[0013] If the number of iterations meets the preset threshold, the output image obtained in the last iteration will be determined as the target output image;

[0014] For each of the above initial signal accumulation training images, a target loss value is generated based on the above target output image and the image label corresponding to the above target output image;

[0015] The network parameters of the initial image processing model are iteratively adjusted based on multiple target loss values ​​to obtain the fluorescence imaging model.

[0016] According to embodiments of this disclosure, the network parameters of the initial image processing model are iteratively adjusted based on multiple target loss values ​​to obtain the fluorescence imaging model, including:

[0017] Generate the total target loss value based on the multiple target loss values ​​mentioned above;

[0018] If the total target loss does not meet the preset loss threshold, the network parameters are adjusted according to the total target loss to obtain the adjusted initial image processing model.

[0019] The target output image is used to determine a new initial signal accumulation training image, and the adjusted initial image processing model is used to perform iterative processing with a preset number of thresholds to obtain the fluorescence imaging model.

[0020] If the total target loss meets the preset loss threshold, the initial image processing model after adjusting the network parameters is determined as the fluorescence imaging model.

[0021] According to embodiments of this disclosure, a new initial signal accumulation training image is determined from the target output image to perform iterative processing with the adjusted initial image processing model for a preset number of times to obtain the fluorescence imaging model, including:

[0022] For each of the above-mentioned new initial signal accumulation training images, the above-mentioned new initial signal accumulation training images are input into the above-mentioned adjusted initial image processing model and iterated for a preset number of times to obtain a new target output image;

[0023] Based on the multiple new target output images and multiple image labels mentioned above, a new total target loss value is generated;

[0024] Based on the new target total loss value and the preset loss threshold, the network parameters of the adjusted initial image processing model are iteratively adjusted to obtain the fluorescence imaging model.

[0025] According to embodiments of this disclosure, the adaptive gated noise reduction network described above is used to process multiple gradient variable parameters to obtain an intermediate output image corresponding to each of the initial signal accumulation training images, including:

[0026] For each of the above gradient variable parameters, the gradient variable parameters are input into the feature extraction network for convolutional encoding to obtain the initial encoded feature map;

[0027] The initial encoded feature map is input into the adaptive memory fusion network to obtain the fused encoded feature map;

[0028] The above integrated coding feature map is input into the residual reconstruction network to obtain the target coding feature map;

[0029] Based on the initial encoded feature map and the target encoded feature map, the intermediate output image is generated.

[0030] According to embodiments of this disclosure, the adaptive memory integration network includes i gated memory integration units;

[0031] The initial encoded feature map is input into the adaptive memory fusion network to obtain the fused encoded feature map, which includes:

[0032] When m=1, the m-th gated memory integration unit is used to process the above initial coding feature map to obtain the i-th intermediate coding feature map;

[0033] When m≠1, the m-th gated memory integration unit processes the initial encoded feature map and the intermediate encoded feature map output by the first i-1 gated memory integration units to obtain the i-th intermediate encoded feature map.

[0034] When m=i, the i-th intermediate coding feature map output by the m-th gated memory integration unit is determined as the integrated coding feature map.

[0035] According to embodiments of this disclosure, the initial encoded feature map and the intermediate encoded feature maps output by the first i-1 gated memory integration units are processed by the m-th gated memory integration unit to obtain the i-th intermediate encoded feature map, including:

[0036] Based on the preset element addition function, a target feature map is generated according to the above initial encoded feature map and multiple above intermediate encoded feature maps;

[0037] Generate affine parameters based on the target feature map described above;

[0038] Based on the feature fusion function, the i-th intermediate encoded feature map is generated according to the above affine parameters and the above target feature map.

[0039] According to embodiments of this disclosure, the preset element addition function is shown in the first formula, and the feature fusion function is shown in the second formula:

[0040]

[0041]

[0042] in, Characterize the initial or intermediate coding feature map. Characterizing the target feature map, and All are affine parameters, and n is the nth gated memory integration unit.

[0043] According to embodiments of this disclosure, the differentiable gradient descent model described above is as shown in the first formula, and the adaptive gated noise reduction network adopts the function model shown in the second formula:

[0044] (3)

[0045] (4)

[0046] in, Accumulate training images for the initial signal. The target output image during the iteration process. For adjustable step size, For gradient variable parameters, For adaptive gated noise reduction networks, The target output image after the iteration is complete.

[0047] Another aspect of this disclosure provides a fluorescence imaging apparatus, comprising:

[0048] The acquisition module is used to acquire fluorescence images of signal accumulation.

[0049] The processing module is used to input the above-mentioned signal accumulation fluorescence image into the trained fluorescence imaging model and output the denoised target fluorescence image. The fluorescence imaging model is constructed based on the differentiable gradient descent model and the adaptive gated denoising network.

[0050] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0051] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0052] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0053] According to embodiments of this disclosure, a fluorescence imaging model constructed using a differentiable gradient descent model and an adaptive gated denoising network can be used to process the acquired signal accumulation fluorescence image to obtain a denoised target fluorescence image. This effectively improves the imaging accuracy of near-infrared fluorescence imaging technology and can be widely applied in fields such as vascular imaging, neural imaging, and tumor boundary imaging. Attached Figure Description

[0054] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0055] Figure 1 An exemplary system architecture to which fluorescence imaging methods can be applied according to embodiments of the present disclosure is illustrated schematically;

[0056] Figure 2 A flowchart illustrating a fluorescence imaging method according to an embodiment of the present disclosure is shown schematically;

[0057] Figure 3 A flowchart illustrating a method for training a fluorescence imaging model according to an embodiment of the present disclosure is shown schematically.

[0058] Figure 4 A schematic diagram illustrating the structure of an initial image processing model according to an embodiment of the present disclosure is shown.

[0059] Figure 5 A schematic comparison diagram of a conventional imaging image and a target fluorescence image according to an embodiment of the present disclosure is shown;

[0060] Figure 6 A block diagram of a fluorescence imaging apparatus according to an embodiment of the present disclosure is shown schematically;

[0061] Figure 7 A block diagram schematically illustrates a training apparatus for a fluorescence imaging model according to an embodiment of the present disclosure; and

[0062] Figure 8 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Detailed Implementation

[0063] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0064] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0065] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0066] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0067] Figure 1 An exemplary system architecture 100 to which fluorescence imaging methods can be applied according to embodiments of this disclosure is illustrated schematically. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0068] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links.

[0069] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0070] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0071] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0072] It should be noted that the fluorescence imaging method provided in this embodiment can generally be executed by server 105. Correspondingly, the fluorescence imaging device provided in this embodiment can generally be located in server 105. The fluorescence imaging method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the fluorescence imaging device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the fluorescence imaging method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Correspondingly, the fluorescence imaging device provided in this embodiment can also be located in terminal devices 101, 102, or 103, or in other terminal devices different from terminal devices 101, 102, or 103.

[0073] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0074] Figure 2 A flowchart illustrating a fluorescence imaging method according to an embodiment of the present disclosure is shown schematically.

[0075] like Figure 2 As shown, the fluorescence imaging method includes operations S201~S203.

[0076] During operation S201, acquire signal accumulation fluorescence images;

[0077] In operation S202, the accumulated fluorescence image is input into the trained fluorescence imaging model, and the denoised target fluorescence image is output. The fluorescence imaging model is constructed based on the differentiable gradient descent model and the adaptive gated denoising network.

[0078] According to embodiments of this disclosure, the signal accumulation fluorescence image can be a fast low signal accumulation fluorescence image, which is an image obtained by capturing the signal emitted by the fluorophore through a near-infrared camera acquisition module at a millisecond gradient. The near-infrared camera acquisition module can also acquire a slow high signal accumulation fluorescence image at a second gradient.

[0079] According to embodiments of this disclosure, the accumulated fluorescence image of the signal can be input into a trained fluorescence imaging model to obtain a denoised target fluorescence image.

[0080] According to embodiments of this disclosure, a fluorescence imaging model constructed using a differentiable gradient descent model and an adaptive gated denoising network can be used to process the acquired signal accumulation fluorescence image to obtain a denoised target fluorescence image. This effectively improves the imaging accuracy of near-infrared fluorescence imaging technology and can be widely applied in fields such as vascular imaging, neural imaging, and tumor boundary imaging.

[0081] Figure 3 A flowchart illustrating a training method for a fluorescence imaging model according to an embodiment of the present disclosure is shown schematically. Figure 4 A schematic diagram of the structure of an initial image processing model according to an embodiment of the present disclosure is shown.

[0082] like Figure 3 As shown, the training method for the fluorescence imaging model includes operations S301~S307.

[0083] In operation S301, a fluorescence image training set is acquired, wherein the fluorescence image training set includes multiple initial signal accumulation training images and image labels corresponding to each initial signal accumulation training image;

[0084] In operation S302, a differentiable gradient descent model is used to process multiple initial signal accumulation training images to obtain gradient variable parameters corresponding to each initial signal accumulation training image.

[0085] In operation S303, an adaptive gated denoising network is used to process multiple gradient variable parameters to obtain an intermediate output image corresponding to each initial signal accumulation training image. The initial image processing model includes a differentiable gradient descent model and an adaptive gated denoising network.

[0086] In operation S304, the intermediate output image is used as a new initial signal to accumulate training images for iterative processing using the initial image processing model.

[0087] In operation S305, if the number of iterations meets the preset threshold, the output image obtained in the last iteration is determined as the target output image;

[0088] In operation S306, training images are accumulated for each initial signal, and a target loss value is generated based on the target output image and the image label corresponding to the target output image.

[0089] In operation S307, the network parameters of the initial image processing model are iteratively adjusted based on multiple target loss values ​​to obtain the fluorescence imaging model.

[0090] According to embodiments of this disclosure, fast low-signal-accumulation fluorescence images and slow high-signal-accumulation fluorescence images of the same size and matching are acquired. The fast low-signal-accumulation fluorescence images are used as high-quality images for data aggregation and integrated into the original domain (i.e., the initial signal accumulation training image) of the near-infrared fluorescence image. The slow high-signal-accumulation fluorescence images are used as high-quality images for data aggregation and integrated into the target domain (i.e., the image label) of the near-infrared fluorescence image.

[0091] According to embodiments of this disclosure, in order to improve the diversity of the dataset, multi-parameter geometric transformations can be performed on the integrated original domain-target domain paired fluorescence images, including any one or more steps such as random rotation transformation, mirror transformation, random cropping, and color distortion, thereby obtaining multiple initial signal accumulation training images.

[0092] According to embodiments of this disclosure, see Figure 4 For each initial signal, accumulate training images and input them into the differentiable gradient descent model shown in formula (1) to obtain the gradient variable parameters. :

[0093] (1)

[0094] in, Accumulate training images for the initial signal. The target output image during the iteration process. For adjustable step size, These are the gradient variable parameters.

[0095] It should be noted that the differentiable gradient descent model described above is obtained by solving the denoising optimization problem shown in equation (2):

[0096] (2)

[0097] Where J is the solution space constraint regularization function, c is the initial signal accumulation training image, and b is the image label corresponding to the initial signal accumulation training image.

[0098] According to embodiments of this disclosure, an adaptive gated noise reduction network as shown in formula (3) is used to process multiple gradient variable parameters to obtain an intermediate output image corresponding to each initial signal accumulation training image. Among them, the adaptive gated noise reduction network These are the gradient variable parameters obtained above. Designed.

[0099] (3)

[0100] According to embodiments of this disclosure, the intermediate output image is used as a new initial signal accumulation training image to perform T rounds of iterative processing using the initial image processing model. When T is greater than or equal to a preset number threshold, the output image obtained in the last iteration is determined as the target output image. At this time, the target loss value during model training is calculated based on the target output image and image label. Based on the target loss value, the network parameters of the initial image processing model can be iterated to obtain the fluorescence imaging model.

[0101] The preset threshold number of times can be set according to actual needs or the precision of the output image; for example, it can be set to 100.

[0102] In another embodiment, the initial learning rate of the image processing model during training can be set to 0.0001, and the Adam optimization algorithm can be used, with its internal parameters set as follows: , .

[0103] According to embodiments of this disclosure, a fluorescence imaging model is obtained by iteratively adjusting the network parameters of an initial image processing model based on multiple target loss values, including:

[0104] Generate a total target loss value based on multiple target loss values;

[0105] If the total target loss does not meet the preset loss threshold, the network parameters are adjusted according to the total target loss to obtain the adjusted initial image processing model.

[0106] The target output image is used to determine a new initial signal accumulation training image to perform iterative processing with a preset threshold number of times using the adjusted initial image processing model, so as to obtain a fluorescence imaging model.

[0107] If the total target loss value meets the preset loss threshold, the initial image processing model after adjusting the network parameters is determined as the fluorescence imaging model.

[0108] According to embodiments of this disclosure, a target loss value is generated for each target output image and its corresponding image label. Multiple target loss values ​​are summed to obtain a total target loss value. If the total target loss value does not meet a preset loss threshold, T iterations of training are performed again until the total target loss value meets the preset loss threshold. The preset loss threshold can be specifically set according to actual needs.

[0109] According to an embodiment of this disclosure, the target loss value is calculated as shown in formula (4):

[0110] (4)

[0111] in, Output an image for a target. Let M be an image label, and M be the number of iterations.

[0112] According to embodiments of this disclosure, a new initial signal accumulation training image is determined from the target output image to perform iterative processing with a preset threshold number of times using an adjusted initial image processing model to obtain a fluorescence imaging model, including:

[0113] For each new initial signal accumulation training image, the new initial signal accumulation training image is input into the adjusted initial image processing model and iterated for a preset number of times to obtain a new target output image;

[0114] Based on multiple new target output images and multiple image labels, a new total target loss value is generated;

[0115] Based on the new target total loss value and the preset loss threshold, the network parameters of the adjusted initial image processing model are iteratively adjusted to obtain the fluorescence imaging model.

[0116] According to embodiments of this disclosure, the new target loss value is obtained by summing the target loss values ​​corresponding to the training images accumulated from a certain initial signal in each round, and the new total target loss value is obtained by summing the new target loss values ​​corresponding to each new total target loss value.

[0117] According to embodiments of this disclosure, referring to Figure 4 An adaptive gated denoising network is used to process multiple gradient variable parameters to obtain intermediate output images corresponding to each initial signal accumulation training image, including:

[0118] For each gradient variable parameter, the gradient variable parameter is input into the feature extraction network for convolutional encoding to obtain the initial encoded feature map;

[0119] The initial encoded feature map is input into the adaptive memory fusion network to obtain the fused encoded feature map;

[0120] The integrated coding feature map is input into the residual reconstruction network to obtain the target coding feature map;

[0121] An intermediate output image is generated based on the initial encoded feature map and the target encoded feature map.

[0122] According to embodiments of this disclosure, the feature extraction network consists of convolutional layers, with the optimized gradient variable parameters from the previous layer as input and the initial encoded feature map as output. The convolutional layers are composed of alternating combinations of double convolutional downsampling units and max-pooling layers to capture multi-scale contextual information and obtain multi-scale feature representations.

[0123] According to embodiments of this disclosure, the dual convolutional downsampling unit includes a convolutional layer, an instance normalization layer, an activation function, and a random neuron deactivation function. The random neuron deactivation function is used to deactivate local neurons, thereby preventing overfitting of the neural network. The encoded feature map is input to the dual convolutional downsampling unit and passes through the convolutional downsampling layer, the instance normalization layer, the activation function layer, and the random neuron deactivation function, respectively. Weighted connections are used between the network layers, and the ReLU activation function is used for non-linear activation, as shown in formula (5).

[0124] (5)

[0125] in This represents the i-th neuron in the n-th layer of the network. Indicates the first The j-th neuron in the layer network, Let represent the connection weight of the i-th neuron, and ReLU represent the activation function as shown in equation (6):

[0126] (6)

[0127] According to embodiments of this disclosure, after the feature extraction network extracts features, the input of the adaptive memory fusion network is the initial encoded feature map, and the output is the fused encoded feature map.

[0128] According to embodiments of this disclosure, the residual reconstruction network consists of convolutional layers. The input is an integrated coding feature map, and the output is a residual map (i.e., a target coding feature map). The convolutional layers are composed of alternating combinations of double convolutional upsampling units and max pooling layers to capture multi-scale contextual information and obtain a residual map representing multi-scale features. The gradient variable parameters are added to the residual map to obtain the output of the initial image processing model, i.e., the intermediate output image after optimization in this round. This intermediate output image maps the input fast low-signal accumulation fluorescence image into a high-quality fluorescence image with the characteristics of a slow high-signal accumulation image.

[0129] According to embodiments of this disclosure, the adaptive memory integration network includes i gated memory integration units.

[0130] According to embodiments of this disclosure, referring to Figure 4 The initial encoded feature map is input into the adaptive memory fusion network to obtain the fused encoded feature map, including:

[0131] When m=1, the initial encoded feature map is processed by the m-th gated memory integration unit to obtain the i-th intermediate encoded feature map;

[0132] When m≠1, the initial encoded feature map and the intermediate encoded feature map output by the first i-1 gated memory integration units are processed by the m-th gated memory integration unit to obtain the i-th intermediate encoded feature map.

[0133] When m=i, the i-th intermediate coding feature map output by the m-th gated memory integration unit is determined as the integrated coding feature map.

[0134] In one specific embodiment, reference is made to... Figure 4 If the adaptive memory ensemble network comprises three gated memory ensemble units, the first gated memory ensemble unit processes the initial encoded feature map output by the feature extraction network to obtain a first intermediate encoded feature map. The second gated memory ensemble unit processes the first intermediate encoded feature map output by the first gated memory ensemble unit and the initial encoded feature map output by the feature extraction network to obtain a second intermediate encoded feature map. The third gated memory ensemble unit processes the first intermediate encoded feature map, the second intermediate encoded feature map, and the initial encoded feature map to obtain a third intermediate encoded feature map. This third intermediate encoded feature map is determined as the ensemble encoded feature map.

[0135] According to embodiments of this disclosure, the initial encoded feature map and the intermediate encoded feature maps output by the first i-1 gated memory integration units are processed using the m-th gated memory integration unit to obtain the i-th intermediate encoded feature map, including:

[0136] Based on a preset element addition function, a target feature map is generated according to the initial encoded feature map and multiple intermediate encoded feature maps;

[0137] Generate affine parameters based on the target feature map;

[0138] Based on the feature fusion function, the i-th intermediate encoded feature map is generated according to the affine parameters and the target feature map.

[0139] According to embodiments of this disclosure, for the gated memory integration unit of the i-th level, the outputs of previous levels are transmitted to the current stage, embedded by independent convolutional layers, and then multiple elements are merged using the preset element addition function shown in formula (7):

[0140] (7)

[0141] in, Characterize the initial or intermediate coding feature map. Characterize the target feature map.

[0142] According to embodiments of this disclosure, the intermediate encoded feature map output at level i is represented as follows: Used to calculate two affine parameters As shown in formula (8):

[0143] (8)

[0144] in, Characterize the initial or intermediate coding feature map. Characterizing the target feature map, and All are affine parameters, and n is the nth gated memory integration unit.

[0145] According to embodiments of this disclosure, the intermediate coding feature map of this level is converted into an information integration coding feature map. This refers to the i-th intermediate encoded feature map, where C, H, and W represent the channel size, height, and width, respectively. The i-th intermediate encoded feature map is the result of feature fusion between the i-th level stages. As shown in formula (9):

[0146] (9)

[0147] According to embodiments of this disclosure, during the training process of the fluorescence imaging model, a validation set and a test set can be set for verification to detect the imaging effect of the fluorescence imaging model.

[0148] Figure 5 A schematic comparison diagram of a conventional imaging image and a target fluorescence image according to an embodiment of the present disclosure is shown.

[0149] In one specific embodiment, this disclosure involves injecting a fluorescent probe into a mouse via the tail vein and then performing fluorescence imaging within a short-wavelength near-infrared window in vivo. This results in a rapid low-signal accumulation fluorescence image of the mouse's blood vessels, which is the signal accumulation fluorescence image of this disclosure. Figure 5 As shown in (a), the fluorescence image with signal accumulation is input into the trained fluorescence imaging model. A high-quality fluorescence image with slow but high signal accumulation characteristics is obtained through a differentiable gradient descent model and an adaptive gated noise reduction network, i.e., as shown in (a). Figure 5 (b) shows the target fluorescence image.

[0150] Figure 6 A block diagram of a fluorescence imaging apparatus according to an embodiment of the present disclosure is shown schematically.

[0151] like Figure 6 As shown, the fluorescence imaging device 600 includes an acquisition module 610 and a processing module 620.

[0152] Acquisition module 610 is used to acquire fluorescence images of signal accumulation;

[0153] The processing module 620 is used to input the signal accumulation fluorescence image into the trained fluorescence imaging model and output the denoised target fluorescence image. The fluorescence imaging model is constructed based on the differentiable gradient descent model and the adaptive gated denoising network.

[0154] According to embodiments of this disclosure, a fluorescence imaging model constructed using a differentiable gradient descent model and an adaptive gated denoising network can be used to process the acquired signal accumulation fluorescence image to obtain a denoised target fluorescence image. This effectively improves the imaging accuracy of near-infrared fluorescence imaging technology and can be widely applied in fields such as vascular imaging, neural imaging, and tumor boundary imaging.

[0155] Figure 7 A block diagram of a training apparatus for a fluorescence imaging model according to an embodiment of the present disclosure is shown schematically.

[0156] like Figure 7 As shown, the training device 700 for the fluorescence imaging model includes a second acquisition module 710, a gradient descent module 720, a noise reduction module 730, an iterative training module 740, a determination module 750, a generation module 760, and an acquisition module 770.

[0157] The second acquisition module 710 is used to acquire a fluorescence image training set, wherein the fluorescence image training set includes multiple initial signal accumulation training images and image labels corresponding to each initial signal accumulation training image;

[0158] The gradient descent module 720 is used to process multiple initial signal accumulation training images using a differentiable gradient descent model to obtain gradient variable parameters corresponding to each initial signal accumulation training image.

[0159] The noise reduction module 730 is used to process multiple gradient variable parameters using an adaptive gated noise reduction network to obtain an intermediate output image corresponding to each initial signal accumulation training image. The initial image processing model includes a differentiable gradient descent model and an adaptive gated noise reduction network.

[0160] The iterative training module 740 is used to accumulate training images by using intermediate output images as new initial signals to perform iterative processing using the initial image processing model.

[0161] The determination module 750 is used to determine the output image obtained in the last iteration as the target output image when the number of iterations meets a preset threshold.

[0162] The generation module 760 is used to accumulate training images for each initial signal and generate target loss values ​​based on the target output image and the image labels corresponding to the target output image.

[0163] Module 770 is obtained, which is used to iteratively adjust the network parameters of the initial image processing model based on multiple target loss values ​​to obtain the fluorescence imaging model.

[0164] According to embodiments of this disclosure, module 770 includes:

[0165] The first generation submodule is used to generate the total target loss value based on multiple target loss values;

[0166] The first submodule is used to adjust the network parameters according to the target total loss value when the target total loss value does not meet the preset loss threshold, so as to obtain the adjusted initial image processing model.

[0167] The iterative submodule is used to determine a new initial signal from the target output image, accumulate training images, and perform iterative processing with a preset threshold number of times using the adjusted initial image processing model to obtain a fluorescence imaging model.

[0168] The determination submodule is used to determine the initial image processing model after network parameter adjustment as a fluorescence imaging model when the total target loss value meets the preset loss threshold.

[0169] According to embodiments of this disclosure, the iterative submodule includes:

[0170] The iterative unit is used to accumulate training images for each new initial signal. The new initial signal accumulated training image is input into the adjusted initial image processing model and iteratively processed for a preset number of times to obtain a new target output image.

[0171] The generation unit is used to generate a new total target loss value based on multiple new target output images and multiple image labels;

[0172] The obtained unit is used to iteratively adjust the network parameters of the adjusted initial image processing model based on the new target total loss value and the preset loss threshold, so as to obtain the fluorescence imaging model.

[0173] According to embodiments of this disclosure, the noise reduction module 730 includes:

[0174] The second submodule is used to input the gradient variable parameters into the feature extraction network for convolutional encoding processing to obtain the initial encoded feature map for each gradient variable parameter.

[0175] The integration submodule is used to input the initial encoded feature map into the adaptive memory integration network to obtain the integrated encoded feature map;

[0176] The residual submodule is used to input the integrated coding feature map into the residual reconstruction network to obtain the target coding feature map;

[0177] The second generation submodule is used to generate intermediate output images based on the initial encoded feature map and the target encoded feature map.

[0178] According to embodiments of this disclosure, the adaptive memory integration network includes i gated memory integration units.

[0179] According to embodiments of this disclosure, the integration submodule includes:

[0180] The first processing unit is used to process the initial encoded feature map using the m-th gated memory integration unit when m=1, to obtain the i-th intermediate encoded feature map;

[0181] The second processing unit is used to process the initial coding feature map and the intermediate coding feature map output by the first i-1 gated memory integration units when m≠1, so as to obtain the i-th intermediate coding feature map.

[0182] The determination unit is used to determine the i-th intermediate coding feature map output by the m-th gated memory integration unit as the integrated coding feature map when m=i.

[0183] According to embodiments of this disclosure, the second processing unit includes:

[0184] The first generation subunit is used to generate a target feature map based on a preset element addition function, according to the initial encoded feature map and multiple intermediate encoded feature maps;

[0185] The second generation subunit is used to generate affine parameters based on the target feature map;

[0186] The third generation subunit is used to generate the i-th intermediate encoded feature map based on the feature fusion function, according to the affine parameters and the target feature map.

[0187] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuits, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented by hardware or firmware through any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in a suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0188] For example, any multiple of the following modules can be implemented in a single module / unit / subunit: acquisition module 610, processing module 620, second acquisition module 710, gradient descent module 720, noise reduction module 730, iterative training module 740, determination module 750, generation module 760, and obtaining module 770. Alternatively, any one of these modules / units / subunits can be split into multiple modules / units / subunits. Or, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in a single module / unit / subunit. According to embodiments of this disclosure, at least one of the acquisition module 610, processing module 620, or second acquisition module 710, gradient descent module 720, noise reduction module 730, iterative training module 740, determination module 750, generation module 760, and obtaining module 770 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 610, processing module 620, or second acquisition module 710, gradient descent module 720, noise reduction module 730, iterative training module 740, determination module 750, generation module 760, and obtaining module 770 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0189] It should be noted that the fluorescence imaging device section in the embodiments of this disclosure corresponds to the fluorescence imaging method section in the embodiments of this disclosure. For a detailed description of the fluorescence imaging device section, please refer to the fluorescence imaging method section, which will not be repeated here. Similarly, the fluorescence imaging model training device section in the embodiments of this disclosure corresponds to the fluorescence imaging model training method section in the embodiments of this disclosure. For a detailed description of the fluorescence imaging model training device section, please refer to the fluorescence imaging model training method section, which will not be repeated here.

[0190] Figure 8 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0191] like Figure 8As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 802 or a program loaded from storage portion 808 into random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0192] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0193] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The system 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0194] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0195] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0196] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0197] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0198] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the training method or fluorescence imaging method of the fluorescence imaging model provided in the embodiments of this disclosure.

[0199] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0200] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0201] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0203] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A fluorescence imaging method, characterized in that, include: Acquire fluorescence images of signal accumulation; The accumulated fluorescence image is input into a trained fluorescence imaging model, and a denoised target fluorescence image is output. The fluorescence imaging model is constructed based on a differentiable gradient descent model and an adaptive gated denoising network. The fluorescence imaging model is trained in the following manner: Obtain a fluorescence image training set, wherein the fluorescence image training set includes multiple initial signal accumulation training images and image labels corresponding to each initial signal accumulation training image; The differentiable gradient descent model is used to process multiple initial signal accumulation training images to obtain gradient variable parameters corresponding to each initial signal accumulation training image; The adaptive gated noise reduction network is used to process multiple gradient variable parameters to obtain an intermediate output image corresponding to each initial signal accumulation training image, wherein the initial image processing model includes the differentiable gradient descent model and the adaptive gated noise reduction network; The intermediate output image is used as a new initial signal to accumulate training images for iterative processing using the initial image processing model. If the number of iterations meets the preset threshold, the output image obtained in the last iteration will be determined as the target output image; For each of the initial signal accumulation training images, a target loss value is generated based on the target output image and the image label corresponding to the target output image; The network parameters of the initial image processing model are iteratively adjusted based on multiple target loss values ​​to obtain the fluorescence imaging model; The process of using the adaptive gated noise reduction network to process multiple gradient variable parameters to obtain an intermediate output image corresponding to each initial signal accumulation training image includes: For each gradient variable parameter, the gradient variable parameter is input into the feature extraction network for convolutional encoding to obtain an initial encoded feature map; The initial encoded feature map is input into the adaptive memory fusion network to obtain the fused encoded feature map; The integrated coding feature map is input into the residual reconstruction network to obtain the target coding feature map; The intermediate output image is generated based on the initial encoded feature map and the target encoded feature map; The adaptive memory integration network includes i gated memory integration units; The initial encoded feature map is input into the adaptive memory fusion network to obtain the fused encoded feature map, including: When n=1, the initial encoded feature map is processed by the nth gated memory integration unit to obtain the nth intermediate encoded feature map; When n≠1, the initial encoded feature map and the intermediate encoded feature maps output by the first n-1 gated memory integration units are processed by the nth gated memory integration unit to obtain the nth intermediate encoded feature map, including: generating a target feature map based on the initial encoded feature map and multiple intermediate encoded feature maps according to a preset element addition function; generating affine parameters according to the target feature map; and generating the nth intermediate encoded feature map based on the feature fusion function, according to the affine parameters and the target feature map. When n=i, the nth intermediate coding feature map output by the nth gated memory integration unit is determined as the integrated coding feature map; The differentiable gradient descent model is shown in Equation (3), and the adaptive gated noise reduction network adopts the function model shown in Equation (4): （3） （4） in, Accumulate training images for the initial signal. The target output image during the iteration process. For adjustable step size, For gradient variable parameters, For adaptive gated noise reduction networks, The target output image after the iteration is complete.

2. The method according to claim 1, characterized in that, The fluorescence imaging model is obtained by iteratively adjusting the network parameters of the initial image processing model based on multiple target loss values, including: Generate a total target loss value based on the multiple target loss values; If the total target loss does not meet the preset loss threshold, the network parameters are adjusted according to the total target loss to obtain the adjusted initial image processing model. The target output image is used to determine a new initial signal accumulation training image, which is then used to perform iterative processing with a preset threshold number of times using the adjusted initial image processing model to obtain the fluorescence imaging model. If the total target loss value meets the preset loss threshold, the initial image processing model after adjusting the network parameters is determined as the fluorescence imaging model.

3. The method according to claim 2, characterized in that, The target output image is used to determine a new initial signal accumulation training image, which is then used to perform iterative processing with a preset threshold number of iterations using the adjusted initial image processing model to obtain the fluorescence imaging model, including: For each new initial signal accumulation training image, the new initial signal accumulation training image is input into the adjusted initial image processing model for iterative processing with a preset number of thresholds to obtain a new target output image; Based on the multiple new target output images and the multiple image labels, a new total target loss value is generated; Based on the new target total loss value and the preset loss threshold, the network parameters of the adjusted initial image processing model are iteratively adjusted to obtain the fluorescence imaging model.

4. The method according to claim 1, characterized in that, The preset element addition function is shown in formula (1), and the feature fusion function is shown in formula (2): （1） （2） in, Characterize the initial or intermediate coding feature map. Characterizing the target feature map, and All are affine parameters, and n is the nth gated memory integration unit.

5. A fluorescence imaging device, employing the method described in any one of claims 1 to 4, characterized in that, include: The acquisition module is used to acquire fluorescence images of signal accumulation. The processing module is used to input the signal accumulation fluorescence image into a trained fluorescence imaging model and output a denoised target fluorescence image, wherein the fluorescence imaging model is constructed based on a differentiable gradient descent model and an adaptive gated denoising network.

Images