Fluorescence image segmentation method based on unsupervised deep learning and self-group iteration upgrading device
By combining unsupervised deep learning with a self-group iterative upgrade device, the problem of poor segmentation results in medical image segmentation is solved, achieving high-accuracy fluorescence image segmentation and avoiding the difficulty of obtaining labeled data and the complex calibration process.
Patent Information
- Application Number
- CN202511236229.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-30
- Publication Date
- 2025-11-21
AI Technical Summary
Existing medical image segmentation methods suffer from problems such as light leakage, reflection, indistinct distinction between background and living body, and noise in areas outside the living body in the image. Furthermore, deep learning-based fluorescence image segmentation lacks sufficient image segmentation annotations.
A fluorescence image segmentation method based on unsupervised deep learning is adopted. Image transformation and label synthesis are performed through a cross-modal medical image segmentation network, and network correction is performed by combining a self-group iterative upgrade device. Online training and correction are carried out using the self-iterative upgrade module and the networked group iterative upgrade module.
It achieves high-accuracy fluorescence image segmentation, avoids the problem of difficulty in obtaining labeled data, and eliminates the need for complex camera calibration and hard threshold parameter tuning, thus improving the segmentation effect.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to a fluorescence image segmentation method based on unsupervised deep learning and a self-group iterative upgrading device. BACKGROUND
[0002] In the field of image processing, compared with visible light and near-infrared region I, the emission wavelength of near-infrared region II is longer, which can significantly reduce the scattering of photons in biological tissues, enhance the light absorption of biological tissues, and penetrate deeper, so it has higher time and spatial resolution, and can more clearly observe the distribution of nerves and blood vessels.
[0003] In fluorescence live imaging, in order to better judge the fluorescence region, image segmentation processing is usually performed. Common image segmentation methods include: mathematical statistics-based methods (gray level histogram), texture-based methods, threshold-based methods, etc.
[0004] Mathematical statistics mainly analyzes the histogram of data, and performs segmentation according to the probability model in the histogram. For example, the proportion of values in a certain range of the image is used to segment the image; or a clustering method is used to cluster the image into different categories.
[0005] The texture-based method mainly segments the image according to the different characteristics (roughness, similarity, etc.) of the object texture, mainly using gray level co-occurrence matrix, fractal theory, histogram matrix, etc. One disadvantage of texture analysis is that it cannot effectively distinguish objects with similar textures but different properties;
[0006] Threshold segmentation usually has two types: empirical threshold and adaptive threshold. The image is segmented by setting a segmentation threshold, and the segmentation effect depends only on the set threshold, the standard is single, and cannot adapt to complex scenes. In actual application, due to insufficient darkroom sealing or unsuitable filters, etc., light leakage, reflection, unclear distinction between background and living body, noise in the area outside the living body in the image and other phenomena often occur, thereby affecting the image segmentation effect.
[0007] Medical image segmentation based on deep learning has achieved near-human-level fully supervised performance. However, in the field of medical imaging, obtaining pixel-level expert annotation is extremely expensive and laborious. SUMMARY
[0008] To solve the problems of light leakage, reflection, unclear distinction between background and living body, noise in the area outside the living body in the image and other problems existing in the traditional image segmentation method in the existing medical image segmentation process, and the lack of sufficient image segmentation annotation in the fluorescence image segmentation method based on deep learning, the present application provides a fluorescence image segmentation method based on unsupervised deep learning. The method comprises:
[0009] The original fluorescence image to be segmented is collected by a fluorescence camera and input into the cross-modal medical image segmentation network to obtain a segmented live fluorescence region.
[0010] In step S100, a series of original fluorescence images are continuously collected by a fluorescence camera as a training set for unsupervised learning.
[0011] In step S200, the training set obtained in step S100 is used to construct and train the cross-modal medical image segmentation network.
[0012] In some preferred embodiments, the training method of the cross-modal medical image segmentation network is as follows:
[0013] In step S210, the original fluorescence image sequence in step S100 is used as a data set to train a generator network based on an intra-slice and inter-slice attention module using an unsupervised learning training mode to perform image conversion.
[0014] In step S220, the trained generator network based on the intra-slice and inter-slice attention module in step S210 is used to convert the two-dimensional original fluorescence image sequence into a three-dimensional image (including time) and synthesize the data label corresponding to the three-dimensional image.
[0015] In step S230, the three-dimensional image data with synthesized labels obtained in step S220 is used as a training data set to train a teacher segmentation network model.
[0016] In step S240, the three-dimensional image data with synthesized labels obtained in step S220 is predicted using the teacher segmentation network model; the synthesized labels of the three-dimensional image data with synthesized labels obtained in step S220 are replaced with the labels predicted by the teacher segmentation network model as a combined data set.
[0017] In step S250, the combined data set obtained in step S240 is used to train a student segmentation network.
[0018] In step S300, the original fluorescence image is input into the cross-modal medical image segmentation network (student segmentation network) to output a live fluorescence region.
[0019] In step S400, the original fluorescence image is input into the cross-modal medical image segmentation network self-group iterative upgrading device to perform online correction on the cross-modal medical image segmentation network.
[0020] The self-group iterative upgrading device comprises a single-machine self-iterative upgrading module and a networked group iterative upgrading module; the difference between self-upgrading and group upgrading lies in whether the training information (data set, data set size, data set quality, etc.) is from a local single machine (self-upgrading) or a networked group (group upgrading); wherein, the group iterative upgrading module comprises two functions: group local iterative upgrading and group cloud iterative upgrading; the difference between group local upgrading and group cloud upgrading lies in whether the training process is completed locally or in the cloud; the training device can be local, in the cloud, or both; the group local iterative upgrading comprises two functions: group local single-machine training iterative upgrading and group local distributed training iterative upgrading; the group local single-machine training iterative upgrading comprises two functions: group local single-machine training shared data set iterative upgrading (without sharing training weights) and group local single-machine training shared data set and weight iterative upgrading.
[0021] The self-iterative upgrading device based on deep learning comprises:
[0022] The storage module is configured to collect the original fluorescence image input into the cross-modal medical image segmentation network.
[0023] The neural network fine-tuning module is configured to correct and fine-tune the cross-modal medical image segmentation network based on the images collected by the storage module.
[0024] The self-iterative upgrading step of the self-iterative upgrading device based on deep learning comprises inputting the original fluorescence image into the cross-modal medical image segmentation network and performing cross-modal medical image segmentation network training.
[0025] The group iterative upgrading device based on deep learning comprises:
[0026] The storage module is configured to collect the original fluorescence image input into the cross-modal medical image segmentation network.
[0027] The neural network fine-tuning module is configured to correct and fine-tune the cross-modal medical image segmentation network based on the images collected by the storage module.
[0028] The network service module is configured to communicate with and transmit data to the cloud control center.
[0029] The group local single-machine training shared data set iterative upgrading based on the group iterative upgrading module comprises: the training data set is from all networked devices; the original fluorescence images shared by several other networked devices are downloaded through the network module and input into the cross-modal medical image segmentation network of the current local device, training is performed locally, the corrected cross-modal medical image segmentation network weights are obtained, and the corrected cross-modal medical image segmentation network weights are used for the current local device.
[0030] The group local single-machine training shared data set and weight iterative upgrading of the group iterative upgrading module comprises: the training data set is from all the networked devices; the prerequisite requirement is that the training model software of all the networked devices is the same; the original fluorescence image of several other networked devices is downloaded to the current local device through a network module and input to the cross-modality medical image segmentation network of the current local device, then training is performed to obtain the corrected cross-modality medical image segmentation network weight, the cross-modality medical image segmentation network weight of the current local device is updated, and is shared in real time to all the networked devices for use by all the networked devices.
[0031] The group local distributed training iterative upgrading based on the group iterative upgrading module comprises: the training information (data set size, data set quality, etc.) is from all the networked devices; the prerequisite requirement is that the training model software of all the networked devices is the same (corresponding to the need to know the data set size of each device) or the quantitative difference of the training data set parameters generated by each device is known (corresponding to the need to know the data set quality of each device); the original fluorescence image of each current local device is input to the cross-modality medical image segmentation network of the current local device, training is performed to obtain the corrected cross-modality medical image segmentation network weight; the corrected cross-modality medical image segmentation network weight of several networked devices is uploaded to the cloud control center for global reduction (AllReduce); if the training model software of all the networked devices is the same, then global reduction is performed according to the data set size of each device, and if the training model software of the networked devices is different and the quantitative difference of the data set parameters of each device is known, then global reduction is performed according to the quantitative difference of the data set quality of each device; the updated cross-modality medical image segmentation network weight is obtained.
[0032] The group cloud iterative upgrading based on the group iterative upgrading module comprises: the prerequisite requirement is that the training model software of all the networked devices and the cloud device is the same; the original fluorescence image of several networked devices is transmitted to the cloud control center through a network service module; the cloud control center collects the received original fluorescence image and performs cross-modality medical image segmentation network training; the local network service module downloads and synchronously updates the cross-modality medical image segmentation network weight of the current device.
[0033] Therefore, compared with the prior art, the present application has the following beneficial effects:
[0034] (1) The present application is based on a deep learning algorithm, and the segmentation process does not require hard threshold parameters and targeted tuning of hyperparameters, nor does it require a complex image machine calibration process;
[0035] (2) The present application is based on a deep learning algorithm, and the fluorescence image segmentation accuracy is higher;
[0036] (3) The application adopts unsupervised learning, achieves similar effect as supervised training, and avoids the problem of difficult acquisition of labeled data. BRIEF DESCRIPTION OF DRAWINGS
[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0038] Figure 1 is a flowchart of an embodiment of a fluorescence image segmentation method based on unsupervised deep learning of the present application.
[0039] Figure 2 is a network structure diagram of a cross-modal medical image segmentation network of the present application.
[0040] Figure 3 is a network structure diagram of a multi-head attention module in the cross-modal medical image segmentation network of the present application.
[0041] Figure 4 is a structural schematic diagram of a self-group iterative upgrade device based on deep learning of the present application.
[0042] Figure 5 is a flowchart of self-iterative upgrade based on deep learning of the present application.
[0043] Figure 6 is a logic block diagram of a self-iterative upgrade module based on deep learning of the present application.
[0044] Figure 7 is a flowchart of group local single machine training shared dataset iterative upgrade based on deep learning of the present application.
[0045] Figure 8 is a logic block diagram of a group local single machine training shared dataset iterative upgrade module based on deep learning of the present application.
[0046] Figure 9 is a flowchart of group local single machine training shared dataset and weight iterative upgrade based on deep learning of the present application.
[0047] Figure 10 is a logic block diagram of a group local single machine training shared dataset and weight iterative upgrade module based on deep learning of the present application.
[0048] Figure 11 is a flowchart of group local distributed training iterative upgrade based on deep learning of the present application.
[0049] Figure 12 is a logic block diagram of the group local distributed iterative upgrading module based on deep learning of the present application.
[0050] Figure 13 is a flowchart of the group cloud iterative upgrading based on deep learning of the present application.
[0051] Figure 14 is a logic block diagram of the group cloud iterative upgrading module based on deep learning of the present application.
[0052] Figure 15 is a network structure diagram of the group local single machine training iterative upgrading based on deep learning of the present application.
[0053] Figure 16 is a network structure diagram of the group local distributed and group cloud iterative upgrading based on deep learning of the present application.
[0054] Figure 17 is a structure diagram of a computer system of a server for implementing the embodiments of the method, system and device of the present application.
[0055] It should be noted that, Figure 15 and Figure 16 are only schematic diagrams, and the number of devices is not limited to 5 or more, and the number of group iterative upgrading devices can be 2, 3 or more. DETAILED DESCRIPTION
[0056] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. It should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and embodiments.
[0057] Referring to Figure 1 , the fluorescence image segmentation method based on unsupervised deep learning of the first embodiment of the present application comprises the following steps:
[0058] The original fluorescence image to be segmented is collected by a fluorescence camera and input into a cross-modal medical image segmentation network to obtain a segmented live fluorescence region.
[0059] In step S100, a series of original fluorescence images are continuously collected by a fluorescence camera as a training set for unsupervised learning.
[0060] Step S200, the unsupervised learning training set obtained in step S100 is used to build and train a cross-modal medical image segmentation network. In some preferred embodiments, the cross-modal medical image segmentation network applied to fluorescence image segmentation is built in the following way:
[0061] Referring to Figure 2 , the cross-modal medical image segmentation network applied to fluorescence image segmentation is sequentially connected by a GE (encoder) module, a multi-head attention module, an orthogonal normalization module, a feedforward neural network, and a GD (decoder) module.
[0062] The GE (encoder) module and the GD (decoder) module are sequentially connected by a multi-layer two-dimensional convolutional neural network.
[0063] The feedforward neural network is a multi-layer fully connected neural network, in which each neuron is connected to the neurons of the previous layer.
[0064] Referring to Figure 3 , the multi-head attention module is connected in parallel by multiple self-attention mechanism modules, and then the outputs of all the self-attention mechanism modules are spliced together and input to a linear layer. The self-attention mechanism module is connected in parallel by three linear layers, and then input to a scaling dot-product attention module.
[0065] In some preferred embodiments, the cross-modal medical image segmentation network is trained in the following way:
[0066] Step S210, using the original fluorescence image sequence in step S100 as a data set, a generator network based on intra-slice and inter-slice attention modules is trained in an unsupervised learning mode to perform image conversion.
[0067] Step S220, using the trained generator network based on intra-slice and inter-slice attention modules in step S210, converting a two-dimensional original fluorescence image sequence into a three-dimensional image (including time), and synthesizing the data label corresponding to the three-dimensional image.
[0068] Step S230, using the three-dimensional image data with synthesized labels obtained in step S220 as a training data set, training a teacher segmentation network model.
[0069] Step S240, using the teacher segmentation network model to predict the three-dimensional image data with synthesized labels obtained in step S220; replacing the synthesized labels of the three-dimensional image data with synthesized labels obtained in step S220 with the labels predicted by the teacher segmentation network model as a combined data set.
[0070] Step S250, using the combined data set obtained in step S240 to train the student segmentation network.
[0071] Step S300, input the original fluorescence image into the cross-modal medical image segmentation network (student segmentation network), and output the obtained live fluorescence region.
[0072] Step S400, input the original fluorescence image into the cross-modal medical image segmentation network self-group iterative upgrade device, and correct the cross-modal medical image segmentation network online.
[0073] Referring to Figure 4 , the self-group iterative upgrade device includes a single computer self-iterative upgrade module and a networked group iterative upgrade module; the difference between self-upgrade and group upgrade is whether the training information (data set, data set size, data set quality, etc.) comes from a local single computer (self-upgrade) or a networked group (group upgrade); wherein, the group iterative upgrade module contains two functions: group local iterative upgrade and group cloud iterative upgrade; the difference between group local upgrade and group cloud upgrade is whether the training process is completed locally or in the cloud; the training device can be local, in the cloud, or both; the group local iterative upgrade contains two functions: group local single computer training iterative upgrade and group local distributed training iterative upgrade; the group local single computer training iterative upgrade contains two functions: group local single computer training shared data set iterative upgrade (without sharing training weights), and group local single computer training shared data set and weight iterative upgrade.
[0074] Referring to Figure 5 , Figure 6 , the self-iterative upgrade device based on deep learning includes:
[0075] The storage module is configured to collect the original fluorescence image input into the cross-modal medical image segmentation network.
[0076] The neural network fine-tuning module is configured to correct and fine-tune the cross-modal medical image segmentation network based on the images collected by the storage module.
[0077] The self-iterative upgrade step based on the self-iterative upgrade device is to input the original fluorescence image into the cross-modal medical image segmentation network and perform cross-modal medical image segmentation network training.
[0078] The group iterative upgrade device based on deep learning includes:
[0079] The storage module is configured to collect the original fluorescence image input into the cross-modal medical image segmentation network.
[0080] A neural network fine-tuning module is configured to correct and fine-tune the cross-modality medical image segmentation network based on the images collected by the storage module.
[0081] A network service module is configured to communicate with and transmit data to the cloud control center.
[0082] Referring to Figure 7 , Figure 8 , Figure 15 The group local single-machine training shared dataset iterative upgrade of the group iterative upgrade module includes: the training dataset is from all the networked devices; the original fluorescence images shared by several other networked devices are downloaded through the network module and input to the current local device cross-modality medical image segmentation network, trained locally to obtain the corrected cross-modality medical image segmentation network weight for use by the current local device.
[0083] Referring to Figure 9 , Figure 10 , Figure 15 The group local distributed training iterative upgrade of the group iterative upgrade module includes: the training information (dataset size, dataset quality, etc.) is from all the networked devices; the prerequisite requirement is that either the training model software of all the networked devices is the same (corresponding to knowing the dataset size of each device) or the quantitative difference of the training dataset parameters generated by each device is known (corresponding to knowing the dataset quality of each device); the original fluorescence images of each current local device are input to the current local device cross-modality medical image segmentation network for training to obtain the corrected cross-modality medical image segmentation network weight; the corrected cross-modality medical image segmentation network weights of several networked devices are uploaded to the cloud control center for global reduction (AllReduce); if the training model software of all the networked devices is the same, then the global reduction is performed according to the dataset size of each device, and if the training model software of the networked devices is different and the quantitative difference of the dataset parameters of each device is known, then the global reduction is performed according to the quantitative difference of the dataset quality of each device; the updated cross-modality medical image segmentation network weight is obtained.
[0084] Referring to Figure 11 , Figure 12 , Figure 16 The group local distributed training iterative upgrade of the group iterative upgrade module includes: the training information (dataset size, dataset quality, etc.) is from all the networked devices; the prerequisite requirement is that either the training model software of all the networked devices is the same (corresponding to knowing the dataset size of each device) or the quantitative difference of the training dataset parameters generated by each device is known (corresponding to knowing the dataset quality of each device); the original fluorescence images of each current local device are input to the current local device cross-modality medical image segmentation network for training to obtain the corrected cross-modality medical image segmentation network weight; the corrected cross-modality medical image segmentation network weights of several networked devices are uploaded to the cloud control center for global reduction (AllReduce); if the training model software of all the networked devices is the same, then the global reduction is performed according to the dataset size of each device, and if the training model software of the networked devices is different and the quantitative difference of the dataset parameters of each device is known, then the global reduction is performed according to the quantitative difference of the dataset quality of each device; the updated cross-modality medical image segmentation network weight is obtained.
[0085] Referring to Figure 13 , Figure 14 , Figure 16 The population cloud iterative upgrade of the population iterative upgrade module comprises the following prerequisites: the training model software of all the networked devices and cloud devices is the same; the original fluorescence images of a plurality of networked devices are transmitted to a cloud control center through a network service module; the cloud control center aggregates the received original fluorescence images and performs cross-modal medical image segmentation network training; and a local network service module downloads and synchronously updates the current device cross-modal medical image segmentation network weight.
[0086] The fluorescence image segmentation system based on unsupervised deep learning of the second embodiment of the application comprises a collection module and a training module.
[0087] The collection module is configured to collect original fluorescence images, input the trained cross-modal medical image segmentation network, and obtain fluorescence images after image segmentation.
[0088] The training method of the cross-modal medical image segmentation network is as follows:
[0089] The training module is configured to obtain fluorescence images after image segmentation.
[0090] The cross-modal medical image segmentation network is constructed and trained by using original fluorescence images collected by a fluorescence camera as a training set.
[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the storage device and the processing device described above and the related descriptions can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.
[0092] It should be noted that the fluorescence imaging system based on the cross-modal medical image segmentation network provided in the foregoing embodiments is only exemplified by the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiments of the application are further divided or combined, for example, the modules of the foregoing embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the application.
[0093] The electronic device of the third embodiment of the present application comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned fluorescence image segmentation method based on unsupervised deep learning.
[0094] The computer readable storage medium of the fourth embodiment of the present application stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned fluorescence image segmentation method based on unsupervised deep learning.
[0095] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the storage device and the processing device described above and the related descriptions can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.
[0096] Those skilled in the art should realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described in the foregoing description. Whether the functions are realized by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0097] Reference will now be made to the following description Figure 17 which shows the structural schematic diagram of a computer system of a server used to implement the method, system and device embodiments of the present application. Figure 17 The server shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.
[0098] As Figure 17As shown, the computer system includes a central processing unit (CPU) 501 which can perform various appropriate actions and processes in accordance with programs stored in a read only memory (ROM) 502 or programs loaded from a storage section 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0099] Connected to the I / O interface 505 are an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as required. A removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 510 as required, so that a computer program read out therefrom is installed in the storage section 508 as required.
[0100] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-described functions defined in the methods of the present disclosure are performed. Note that the computer readable medium of the present disclosure can be either a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0101] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).
[0102] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.
[0103] The terms "first", "second", etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence among the objects.
[0104] The terms "include," "includes" or "including" are intended to be non- limiting inclusive (i.e., for the purpose of open ended claims, that "including" means including without limitation).
[0105] The technical scheme of the present application has been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical schemes after the changes or replacements will all fall within the protection scope of the present application.
Claims
1. A method for fluorescent image segmentation based on unsupervised deep learning, characterized in that, The method is: Step S100, a series of original fluorescence images are continuously collected by a fluorescence camera as a training set for unsupervised learning; Step S200, the training set for unsupervised learning obtained in step S100 is used to construct and train a cross-modal medical image segmentation network; Step S300, the original fluorescence image is input into the cross-modal medical image segmentation network (student segmentation network) to output a live fluorescence region; Step S400, the original fluorescence image is input into the cross-modal medical image segmentation network self-group iterative upgrading device to correct the cross-modal medical image segmentation network online. 2.The unsupervised deep learning based fluorescent image segmentation method according to claim 1, wherein, The cross-modal medical image segmentation network applied to fluorescence image segmentation has the following construction method: Referring to FIG. 2, the cross-modal medical image segmentation network applied to fluorescence image segmentation is sequentially connected by a GE encoder module, a multi-head attention module, an orthogonal normalization module, a feedforward neural network, and a GD decoder; The GE encoder module and the GD decoder module are sequentially connected by a multi-layer two-dimensional convolutional neural network; The feedforward neural network is a multi-layer fully connected neural network, and each neuron is connected only to the neurons of the previous layer; Referring to FIG. 3, the multi-head attention module is connected in parallel by multiple self-attention mechanism modules, and then the outputs of all self-attention mechanism modules are spliced together and input into a linear layer; wherein the self-attention mechanism module is connected in parallel by three linear layers, and then input into a scaling dot product attention module. 3.The unsupervised deep learning based fluorescent image segmentation method of claim 1, wherein, The cross-modal medical image segmentation network applied to fluorescence image segmentation has the following training method: Step S210, using the original fluorescence image sequence in step S100 as a data set, using an unsupervised learning training mode, training a generator network based on intra-slice and inter-slice attention modules to perform image conversion; Step S220, using the trained generator network based on intra-slice and inter-slice attention modules in step S210, converting a two-dimensional original fluorescence image sequence into a three-dimensional image (including time), and synthesizing the data label corresponding to the three-dimensional image; Step S230, using the three-dimensional image data with synthesized labels obtained in step S220 as a training data set, training a teacher segmentation network; Step S240, using the teacher segmentation network to predict the three-dimensional image data with synthesized labels obtained in step S220; replacing the synthesized labels of the three-dimensional image data with synthesized labels obtained in step S220 with the labels predicted by the teacher segmentation network as a combined data set; Step S250, using the combined data set obtained in step S240 to train a student segmentation network.
4. A self-group iterative upgrading device based on deep learning, characterized in that, The self-group iterative upgrading device comprises a single computer self-iterative upgrading module and a networked group iterative upgrading module; the difference between self-upgrading and group upgrading lies in whether the training information (data set, data set size, data set quality, etc.) is from a local single computer (self-upgrading) or a networked group (group upgrading); wherein, the group iterative upgrading module comprises two functions: group local iterative upgrading and group cloud iterative upgrading; the difference between group local upgrading and group cloud upgrading lies in whether the training process is completed locally or in the cloud; the training device can be local, in the cloud, or both; the group local iterative upgrading comprises two functions: group local single computer training iterative upgrading and group local distributed training iterative upgrading; the group local single computer training iterative upgrading comprises two functions: group local single computer training shared data set iterative upgrading (without sharing training weight) and group local single computer training shared data set and weight iterative upgrading.
5. The deep learning based self-iterative upgrading module according to claim 4, wherein, The self-iterative upgrading module based on deep learning comprises: a storage module for collecting the original fluorescence image input into the cross-modal medical image segmentation network; a neural network fine-tuning module for correcting and fine-tuning the cross-modal medical image segmentation network based on the images collected by the storage module; The self-iterative upgrading step based on the self-iterative upgrading device is to input the original fluorescence image into the cross-modal medical image segmentation network for cross-modal medical image segmentation network training. 6.The deep learning based group iterative upgrade module according to claim 4, wherein, The group iterative upgrading module based on deep learning comprises: a storage module for collecting the original fluorescence image input into the cross-modal medical image segmentation network; a neural network fine-tuning module for correcting and fine-tuning the cross-modal medical image segmentation network based on the images collected by the storage module; a network service module for communicating and data transmitting with other devices or cloud control center.
7. The group local single-machine training shared dataset iterative upgrade function of the group iterative upgrade module based on deep learning according to claim 4, characterized in that, The group local single computer training shared data set iterative upgrading based on the group iterative upgrading module comprises: the training data set is from all networked devices; the original fluorescence images shared by several other networked devices are downloaded through the network module and input into the cross-modal medical image segmentation network of the current local device for local training to obtain the corrected cross-modal medical image segmentation network weight for use by the current local device.
8. The group local single machine training shared dataset and weight iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 4, wherein, The group local single computer training shared data set and weight iterative upgrading based on the group iterative upgrading module comprises: the training data set is from all networked devices; the precondition requires that the training model software of all networked devices is the same; the original fluorescence images of several other networked devices are downloaded to the current local device through the network module and input into the cross-modal medical image segmentation network of the current local device, then trained to obtain the corrected cross-modal medical image segmentation network weight, update the cross-modal medical image segmentation network weight of the current local device, and share in real time to all networked devices for use by all networked devices.
9. The group local distributed training iterative upgrade function of the group iterative upgrade module based on deep learning according to claim 4, characterized in that, The group local distributed training iterative upgrading based on the group iterative upgrading module comprises: the training information (data set size, data set quality, etc.) is from all networked devices; The premise requires that either all networked devices have the same training model software (corresponding to the need to know the data set size of each device) or that the quantitative differences in the training data set parameters generated by each device are known (corresponding to the need to know the quality of each device data set); input the original fluorescence image of each current local device into the current local device cross-modality medical image segmentation network for training to obtain the corrected cross-modality medical image segmentation network weight; upload the corrected cross-modality medical image segmentation network weight of a plurality of networked devices to the cloud control center for global reduction (AllReduce); if all networked devices have the same training model software, then global reduction is performed according to the data set size of each device, and if the training model software of the networked devices is different and the quantitative differences in the data set parameters of each device are known, then global reduction is performed according to the quantitative differences in the data set quality of each device; obtain the updated cross-modality medical image segmentation network weight. 10.The group cloud iterative upgrade function of the group iterative upgrade module based on deep learning of claim 4, wherein, The group cloud iterative upgrade based on the group iterative upgrade module comprises: The premise requires that all networked devices and cloud devices have the same training model software; The original fluorescence image of a plurality of networked devices is transmitted to the cloud control center through the network service module; the cloud control center collects the received original fluorescence image and performs cross-modality medical image segmentation network training; the local network service module downloads and synchronously updates the current device cross-modality medical image segmentation network weight.