A label-free beta-amyloid plaque recognition method, device and equipment

Abstract

The present application relates to the technical field of biological image processing, and discloses a label-free beta-amyloid plaque recognition method, device and equipment, which comprises the following steps: collecting TPEF and CARS images of mouse brain coronal sections to obtain a label file by plaque labeling; using bimodal CARS / TPEF as a mask, generating a bimodal label file by using the labeling information in the label file; training a preset neural network by using the bimodal image and the label file to obtain a model meeting preset requirements as a bimodal target recognition network model; inputting the CARS and TPEF images to be recognized into the target detection network model to output the spatial position, pathological type and corresponding confidence of the plaque. The present application uses the features of the target region in composition and morphology as the classification basis, adopts a composite labeling method to classify the sample, can consistently, quantitatively and accurately measure the beta-amyloid plaque, and makes the research work more standardized and intelligent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biological image processing, and in particular to a label-free amyloid beta plaque recognition method, device and equipment. BACKGROUND

[0002] Amyloid beta-protein (Aβ) plaque is one of the significant pathological markers of Alzheimer's disease (AD), mainly representing the persistent accumulation of proteins composed of Aβ. According to the morphology of Aβ aggregation, Aβ plaque can be divided into three pathological types, namely cerebral amyloid angiopathy, core-type and diffuse-type plaque. Accurate positioning and recognition of Aβ plaque helps to understand the pathophysiological mechanism of AD.

[0003] At present, the application of optical microscopes allows the detection of disease biomarkers at the cellular or molecular level. The methods for detecting Aβ plaques using optical microscopes mainly include labeled and label-free methods. The existing technology has the following problems or shortcomings:

[0004] First, the label-free plaque detection method. The label-free detection method relies on the local or global information of the optical microscopic image, and uses traditional image processing methods (such as threshold segmentation) to realize plaque segmentation. Although this method can detect Aβ plaques by extracting intensity information from optical images, due to the uneven distribution of image intensity, and in most cases the large difference in sample slices obtained from different mouse brain tissues, it is difficult to accurately segment the plaques using fixed traditional image processing algorithms.

[0005] Second, the labeled plaque pathological type recognition method. There are mainly two forms of labeled plaque pathological type recognition method; the first form is to perform immunohistochemical staining on the sample, and then identify the pathological type according to the fluorescence characteristics of the target object. The second form is to perform tissue staining on the sample, and to realize plaque pathological type recognition according to the morphological characteristics of the target object. Both of these two forms of identification methods need to stain the sample, and use manual counting to perform statistics and classification, which not only requires a large amount of labor and time cost, but also is easily affected by subjective factors.

[0006] Third, plaque recognition method based on supervised deep learning. Since manual counting is very time-consuming, a deep learning network model is used to learn the relevant features of different pathological types of Aβ plaques in the stained mouse brain slices. Due to the bias of different experts in data labeling, the performance of the deep learning network model for the actual Aβ plaque pathological type recognition task may not be ideal. SUMMARY

[0007] Therefore, the application provides a label-free beta-amyloid plaque recognition method, device and equipment to solve the low recognition result accuracy of beta-amyloid plaques in the prior art.

[0008] In a first aspect, the application provides a label-free beta-amyloid plaque recognition method, comprising:

[0009] Collecting TPEF images and CARS images of APP / PS1 mouse brain coronal sections with a preset thickness, wherein the TPEF images contain spontaneous fluorescence signals of endogenous fluorophores, and the CARS images contain spectral signals;

[0010] Labeling different beta-amyloid plaques in the TPEF images and the CARS images using a preset labeling tool to obtain a labeling file;

[0011] Using the CARS / TPEF dual-mode images as a mask, obtaining labeled images according to the coordinates and types in the labeling file, and reviewing the labeling after traversing all the labeling data in the labeling file to obtain labeled images representing the positions of plaques of different pathological types;

[0012] Dividing the three-channel CARS / TPEF dual-mode images and the corresponding labeled images into a training set and a validation set according to a preset ratio, training a preset neural network based on the training set, verifying the trained model based on the validation set, and obtaining a model meeting preset requirements as a dual-mode target recognition network model;

[0013] Inputting the CARS images and the TPEF images to be recognized into the dual-mode target recognition network model to output the spatial positions, pathological types and corresponding confidence of the beta-amyloid plaques.

[0014] The label-free beta-amyloid plaque recognition method provided in this embodiment uses the features of the target region in composition and morphology as the recognition basis, adopts a composite labeling method, and labels the plaque pathological types by using the differences in morphology and composition of the target region in different modal images, which can not only reduce the difficulty of plaque labeling, but also greatly improve the accuracy of labeling. Using TPEF / CARS dual-mode images instead of traditional single-mode optical images as the input of the deep convolutional network model increases the dimension of network optical information extraction, thereby enabling more consistent, quantitative and accurate measurement of Aβ plaques, and making the research work more standardized and intelligent.

[0015] In an alternative embodiment, the method further comprises: storing the result output by the bimodal target recognition network model as an RGB image with a marked box, a confidence and a category, or a txt file containing the spatial position of the marked box, the confidence and the category information, to show the final recognition and positioning information of the beta-amyloid plaque.

[0016] In an alternative embodiment, the beta-amyloid plaques in the TPEF image and the CARS image are respectively labeled by using a preset labeling tool to form a labeling file, which comprises:

[0017] The LabelImg software is used to label the core plaque based on the 2850 cm -1 CARS image representing the lipid composition of the biological tissue, and the spatial position and size of the core plaque are recorded and saved as a first labeling file;

[0018] The CAA plaque type and the non-CAA plaque type are labeled based on the spatial morphological features in the TPEF image, and the spatial position and size of the CAA plaque type and the non-CAA plaque type are recorded and saved as a second labeling file;

[0019] The CARS image and the TPEF image are combined into a three-channel RGB image to form a CARS / TPEF bimodal image, the category and the coordinate of each labeling data in the first labeling file and the second labeling file are positioned in the CARS / TPEF bimodal image, and the type of the non-CAA plaque type region is determined: if there is only a TPEF image in the corresponding spatial position, the labeling of the non-CAA plaque region is modified to be a diffuse plaque, otherwise the labeling of the region is modified to be a core plaque, the coordinate of each labeling data in the CARS / TPEF bimodal image is positioned in the beta-amyloid plaque immunofluorescence image, and the corresponding region is enlarged to review the labeling, and a final bimodal labeling file is obtained.

[0020] The traditional single mask labeling method has high labeling difficulty and labeling accuracy problems. In the embodiment of the present application, a composite labeling method is used to label the plaque pathological type by using the morphological and component differences of the target region in different modal images, which can not only reduce the difficulty of plaque labeling, but also greatly improve the accuracy of labeling.

[0021] In an alternative embodiment, in the labeling process of the CARS image and the TPEF image, the plaque is selected by a keyboard and mouse input method to create a dictionary dict for query and store the labeled label and the spatial position and size of the marked box into the memory, and saved as an xml format file.

[0022] By storing the CARS image and the TPEF image in a queryable manner, subsequent further type judgment is facilitated.

[0023] In an optional embodiment, before the TPEF image is labeled, the method further comprises: pre-processing the TPEF image by using an image enhancement algorithm to strengthen the characteristics of the plaque with strong autofluorescence.

[0024] The embodiment of the present application pre-processes the TPEF image by using an image enhancement algorithm to strengthen the characteristics of the plaque with strong autofluorescence, thereby reducing the missed detection rate of the plaque.

[0025] In an optional embodiment, the R channel of the combined three-channel RGB image is the CARS image, the G channel is the TPEF image, and the B channel is the image obtained by subtracting the mean value of the TPEF image from the TPEF image; the labeled image is generated in the CARS / TPEF dual-mode image according to the label type and the marking box parameters in each label file, and is saved as an xml format file, which records the spatial position and type of the core type, diffuse type and CAA type beta-amyloid plaque.

[0026] In the embodiment of the present application, the B channel is set as the image obtained by subtracting the mean value of the TPEF image from the TPEF image, which can enhance the autofluorescence information of the plaque region, is beneficial to the differentiation of types, and the annotation in the CARS / TPEF image is reviewed by using the immunofluorescence image of the adjacent slice, which is beneficial to the improvement of recognition accuracy.

[0027] In an optional embodiment, the preset neural network is a target recognition network model of YOLOv4, and the training parameters are as follows: the number of iterations epochs is between 100 and 500, the optimizer uses the Adam optimization algorithm, the loss function uses the complete intersection over union loss and the binary cross-entropy loss, which are respectively used for rectangular box loss, confidence loss and classification loss calculation, the learning rate is between 0.001 and 0.0001, the exponential decay rate of the first moment estimation is 0.9, the exponential decay rate of the second moment estimation is 0.999, and the decay rate is 0.

[0028] The target recognition network model of YOLOv4 and the corresponding training parameters provided by the embodiment of the present application can obtain a dual-mode target recognition network model with high recognition rate.

[0029] In a second aspect, the present application provides a label-free beta-amyloid plaque recognition device, which comprises:

[0030] An image acquisition module is configured to acquire TPEF images and CARS images of APP / PS1 mouse brain coronal sections with a preset thickness, wherein the TPEF images contain autofluorescence signals of endogenous fluorophores, and the CARS images contain spectral signals.

[0031] A label file acquisition module is configured to label different beta-amyloid plaques in the TPEF images and the CARS images respectively by using a preset labeling tool to obtain label files.

[0032] A dual-mode label file acquisition module is configured to obtain labeled images representing positions of plaques of different pathological types by using the CARS / TPEF dual-mode images as a mask, obtaining coordinates and types in the label files, and reviewing the labels after traversing all the label data in the label files.

[0033] A target recognition network model acquisition module is configured to divide the three-channel CARS / TPEF dual-mode images and the corresponding label files into a training set and a verification set according to a preset ratio, train a preset neural network based on the training set, verify the trained model based on the verification set, and obtain a model meeting preset requirements as a dual-mode target recognition network model.

[0034] A plaque recognition module is configured to input a CARS / TPEF image to be recognized into the dual-mode target recognition network model, and output spatial positions, pathological types, and corresponding confidence levels of beta-amyloid plaques.

[0035] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the beta-amyloid plaque recognition method of the first aspect or any of the corresponding embodiments thereof.

[0036] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the beta-amyloid plaque recognition method of the first aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0038] Figure 1is a flowchart of a beta-amyloid plaque recognition method of an embodiment of the present application.

[0039] Figure 2 is a schematic diagram of a target recognition network of YOLOv4 provided by an embodiment of the present application.

[0040] Figure 3 is a schematic diagram of an SPP unit structure of an embodiment of the present application.

[0041] Figure 4 is a structural block diagram of a beta-amyloid plaque recognition device according to an embodiment of the present application.

[0042] Figure 5 is a hardware structure schematic diagram of a computer device of an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] According to an embodiment of the present application, a label-free beta-amyloid plaque recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0045] In this embodiment, a label-free beta-amyloid plaque recognition method is provided. The label-free detection method relies on the autofluorescence signal of the endogenous fluorophore and the vibrational spectral signal of the inherent molecule in the sample, which can be used in a computer device terminal, Figure 1 is a flowchart of a beta-amyloid plaque recognition method according to an embodiment of the present application, as shown in Figure 1 the flowchart includes the following steps:

[0046] In step S101, TPEF images and CARS images of APP / PS1 mouse brain coronal sections of a predetermined thickness are collected. The TPEF images contain autofluorescence signals of endogenous fluorophores, and the CARS images contain spectral signals.

[0047] The embodiment of the application adopts the APP / PS1 mouse brain coronal section with a thickness of 8 μm to record the TPEF (two-photon excited fluorescence) image of the mouse model brain tissue by using a two-photon fluorescence microscope, and to obtain the CARS (coherent anti-Stokes raman scattering) image by using a coherent Stokes Raman scattering microscope, so as to record the spatial distribution of the lipid in the Aβ plaque, and to lay a foundation for the identification and positioning of the pathological type of the Aβ plaque, which is only an example, and is not limited thereto.

[0048] In step S102, the preset marking tool is used to mark different β-amyloid plaques in the TPEF image and the CARS image respectively, so as to obtain a marking file.

[0049] In the embodiment of the application, the LabelImg software is used to mark the core type plaque based on the 2850 cm -1 The CARS image is used to mark the core type plaque, and the spatial position and size of the core type plaque are recorded, and saved as a first marking file. In a specific embodiment, if a plaque is found at a certain position in the CARS image, the plaque is selected by using the mouse input mode to create a dictionary dict (an unordered sequence used for storing data) for query, and the label (such as Cord representing the core type plaque) of the marking and the spatial position and size of the marking box are stored in the memory, and after the image is marked, the image is saved as an xml format file (including the CARS image and the marking data).

[0050] In the embodiment of the application, before the TPEF image is marked, the image enhancement algorithm such as Gamma transformation is used to pre-process the TPEF image, so as to strengthen the characteristics of the plaque with strong spontaneous fluorescence, and to reduce the missing detection rate of the plaque. The CAA (cerebral amyloid angiopathy) plaque type and the non-CAA plaque type are marked based on the spatial morphological characteristics in the TPEF image, and the spatial position and size of the CAA plaque type and the non-CAA plaque type are recorded, and saved as a second marking file. In a specific embodiment, if a plaque is found at a certain position in the TPEF image, according to the spatial morphological characteristics, the label (such as CAA representing the cerebral amyloid angiopathy, and other representing the non-CAA plaque) of the marking is stored by using the mouse input mode, the plaque is selected by using the mouse input mode to create a dictionary dict for query, and the label of the marking and the spatial position and size of the marking box are stored in the memory, and after the image is marked, the image is saved as an xml format file (including the TPEF image and the marking data). When marking, the standard can be appropriately controlled to accurately mark as much as possible, so as to avoid introducing false positive plaques.

[0051] Step S103, using the CARS / TPEF dual-mode image as a mask, obtaining the labeled image according to the coordinates and types in the annotation file, and after traversing all the annotation data in the annotation file, reviewing the annotation to obtain the labeled image representing the positions of plaques of different pathological types.

[0052] The specific process is: combining the CARS image and the TPEF image into a three-channel RGB image to form a CARS / TPEF dual-mode image, the G channel is the TPEF image, and the B channel is the image obtained by subtracting the mean value of the TPEF image from itself; positioning the coordinates of each annotation data in the first annotation file and the second annotation file to the RGB image, and judging the pathological type of the non-CAA plaque type region: if only the TPEF image exists at the corresponding spatial position, the region is labeled as a diffuse plaque, for example, diffuse plaques are represented by Diffuse (diffuse plaques), and otherwise, cored plaques are represented by Cord (cored plaques). After the judgment is completed, the coordinates of each annotation data in the RGB image are positioned to the beta-amyloid plaque immunofluorescence image of the adjacent slice, and the corresponding region is enlarged, so as to review the annotation to obtain the final dual-mode annotation file, and the final annotation file is stored as an xml format file.

[0053] In the embodiment of the present application, a composite annotation method is adopted, and the pathological type of the plaque is annotated by using the shape and composition difference of the target region in different modal images, which can not only reduce the difficulty of plaque marking, but also greatly improve the accuracy of annotation. It is worth noting that in the embodiment of the present application, the beta-amyloid plaque immunofluorescence image of the adjacent slice is used to annotate the dual-mode image, which is an optional operation.

[0054] Step S104, dividing the CARS / TPEF dual-mode image and the corresponding annotation file into a training set and a verification set according to a preset ratio, training a preset neural network based on the training set, verifying the trained model based on the verification set, and obtaining a model meeting a preset requirement as a dual-mode target recognition network model.

[0055] In the embodiment of the present application, the obtained three-channel CARS / TPEF dual-mode image and the corresponding annotation file are used to form a data pair, which is divided into a training set and a verification set in a ratio of 8:2. A target recognition network based on YOLOv4 is built, and the network structure is as follows: Figure 2As shown, it consists of four main parts: Input, Backbone, Neck and Prediction. Input is the input end of the network; the Backbone part is composed of bottom units CBM and CSPX. The CBM unit is composed of a convolution layer (Conv layer), a normalization layer (BN layer) and a Mish activation function. The CSPX unit contains a Conv layer with a step of 2 and a convolution kernel size of 3x3 to realize the down-sampling of the image, followed by a 1x1 size Conv layer and the stacking of X residual components (Res unit). The CBL and SPP units together constitute the Neck part of the network model. The CBL is composed of a Conv layer, a BN layer and a Leaky ReLU activation function. The structure of the SPP unit is as shown: Figure 3 The Prediction part is composed of bottom units CBL and Conv layer. The actual application network is written in Python language and Pytorch architecture, and is only used as an example, not limited thereto.

[0056] During training, training images and training targets are read from the training set and converted into tensor data as training data, which is input into the network for training. The training parameters are: the number of iterations epochs is between 100 and 500 times, the optimizer uses the Adam optimization algorithm, the loss function uses the complete IOU loss (Complete-IOU Loss) and the binary cross entropy loss (Binary Cross Entropy Loss), which are used for rectangular frame loss, confidence loss and classification loss calculation respectively, the learning rate (lr, learning rate) is between 0.001 and 0.0001, the exponential decay rate of first moment estimation beta1 is 0.9, the exponential decay rate of second moment estimation beta2 is 0.999, and the decay rate weight decay is 0. After training, the validation images and target images representing the positions of plaques of different pathological types are read from the validation set, converted into tensor data as validation data, and input into the trained network, and finally a model meeting the preset requirements is obtained as a dual-mode target recognition network model.

[0057] The embodiment of the present application uses the bimodal TPEF / CARS image to replace the traditional single-mode optical image as the input of the deep convolution network model, increases the optical information dimension of the beta-amyloid plaque, and improves the optical information utilization rate. For the existing plaque pathological type recognition method based on deep learning, only morphological information can be used, and there is a personalized annotation problem. The embodiment of the present application takes the features of the target region in composition and morphology as the classification basis, adopts a composite labeling method to classify the sample, so that the beta-amyloid plaque can be measured more consistently, quantitatively and accurately, and the research work is more standardized.

[0058] In step S105, the CARS image and the TPEF image to be recognized are input into the bimodal target recognition network model, and the spatial position, pathological type and corresponding confidence of the beta-amyloid plaque are output.

[0059] The embodiment of the present application stores the result output by the bimodal target recognition network model as an RGB image with a marking box, a confidence and a category, or a txt file containing the marking box position, the confidence and the category information, and displays the final beta-amyloid plaque recognition and positioning information for reference.

[0060] The method provided by the embodiment of the present application takes the features of the target region in composition and morphology as the classification basis, adopts a composite labeling method, and labels the plaque pathological type by using the morphological and compositional differences of the target region in different modal images. The method can not only reduce the difficulty of plaque labeling, but also greatly improve the labeling accuracy. The bimodal TPEF / CARS image is used to replace the traditional single-mode optical image as the input of the deep convolution network model, the optical information dimension of the beta-amyloid plaque is increased, and the optical information utilization rate is improved, so that the beta-amyloid plaque can be measured more consistently, quantitatively and accurately, and the research work is more standardized and intelligent.

[0061] In the embodiment, a label-free beta-amyloid plaque recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0062] The embodiment provides a label-free beta-amyloid plaque recognition device, as shown in Figure 4 , comprising:

[0063] The image acquisition module 401 is configured to acquire TPEF images and CARS images of APP / PS1 mouse brain coronal sections with a preset thickness, wherein the TPEF images contain autofluorescence signals of endogenous fluorophores, and the CARS images contain spectral signals.

[0064] The annotation file acquisition module 402 is configured to use a preset annotation tool to respectively annotate different beta-amyloid plaques in the TPEF images and the CARS images, to obtain annotation files.

[0065] The dual-mode annotation file acquisition module 403 is configured to use the CARS / TPEF dual-mode images as a mask, to acquire annotated images according to coordinates and types in the annotation files, to review the annotations after traversing all the annotation data in the annotation files, and to obtain annotated images representing positions of plaques of different pathological types.

[0066] The target recognition network model acquisition module 404 is configured to divide the three-channel CARS / TPEF dual-mode images and corresponding annotated images into a training set and a verification set according to a preset ratio, to train a preset neural network based on the training set, to verify the trained model based on the verification set, and to obtain a model meeting preset requirements as a dual-mode target recognition network model.

[0067] The plaque recognition module 405 is configured to input CARS images and TPEF images to be recognized into the dual-mode target recognition network model, and to output spatial positions, pathological types and corresponding confidence of beta-amyloid plaques.

[0068] The beta-amyloid plaque recognition device in the embodiment is presented in the form of functional units, wherein the units refer to ASIC circuits, processors and memories executing one or more software or fixed programs, and / or other devices capable of providing the above functions.

[0069] Further function descriptions of the above modules and units are the same as those of the corresponding embodiments, and will not be described here.

[0070] The embodiment of the present application also provides a computer device with the above Figure 4 The beta-amyloid plaque recognition device.

[0071] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of a computer device provided by an optional embodiment of the present application, as shown in Figure 5As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for the various components. The various components communicate over one or more busses (constituting a bus system) 40 and can be mounted on a common motherboard or in other manners as appropriate. The processor 10 can process instructions for execution within the computer device, including instructions stored in the memory 20 or elsewhere or instructions fetched from an external source or storage medium such as a display device coupled to the interface 30. In some embodiments, multiple processors and / or multiple buses can be employed as appropriate, along with multiple memories and types of memory. Also, multiple computer devices can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or multiple processors). Figure 5 The processor 10 is taken as an example in the embodiments.

[0072] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0073] The memory 20 stores instructions that are executable by the at least one processor 10, so as to enable the at least one processor 10 to perform the method shown in the above embodiments.

[0074] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory that is remotely arranged with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0075] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned types of memories.

[0076] The computer device further includes a communication interface 30 for communication of the computer device with other devices or communication networks.

[0077] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer codes stored in a remote storage medium or a non-transitory machine readable storage medium and stored in a local storage medium to be downloaded through a network, so that the method described herein can be processed by such software on a storage medium using a general computer, a special processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer codes, and when the software or computer codes are accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0078] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for identifying label-free β-amyloid plaques, characterized in that, The method includes: TPEF and CARS images of coronal sections of APP / PS1 mouse brains of a predetermined thickness were acquired. The TPEF image contained autofluorescence signals of endogenous fluorophores, and the CARS image contained spectral signals. Different β-amyloid plaques were annotated on TPEF and CARS images using a preset annotation tool to obtain annotation files; Using CARS / TPEF bimodal images as masks, labeled images are obtained according to the coordinates and types in the labeled files. After traversing all the labeled data in the labeled files, the labeled images are reviewed to obtain labeled images representing the locations of plaques of different pathological types. The CARS / TPEF bimodal images and their corresponding labeled images are divided into training and validation sets according to a preset ratio. The preset neural network is trained based on the training set, and the trained model is validated based on the validation set. The model that meets the preset requirements is used as the bimodal target recognition network model. The CARS and TPEF images to be identified are input into a bimodal target recognition network model, which outputs the spatial location, pathological type, and corresponding confidence level of the β-amyloid plaque.

2. The method according to claim 1, characterized in that, Also includes: The output of the bimodal target recognition network model is stored as an RGB image with bounding boxes, confidence scores, and categories, or as a txt file containing the spatial location, confidence scores, and category information of the bounding boxes, to demonstrate the final identification and localization information of β-amyloid plaques.

3. The method according to claim 2, characterized in that, The step involves using a preset annotation tool to annotate different β-amyloid plaques on TPEF and CARS images, respectively, to create annotation files, including: LabelImg software was used to characterize the lipid composition of biological tissues at a depth of 2850 cm. -1 CARS images are used to label core-type patches, and the spatial location and size of the core-type patches are recorded and saved as the first annotation file; Based on the spatial morphological features in the TPEF image, CAA patch types and non-CAA patch types are labeled, and the spatial location and size of CAA and non-CAA patch types are recorded and saved as a second annotation file; The CARS and TPEF images are merged into a three-channel RGB image to form a CARS / TPEF bimodal image. The category and coordinates of each annotation data in the first and second annotation files are located in the CARS / TPEF bimodal image, and the type of non-CAA plaque type regions is determined: if the corresponding spatial location only exists in the TPEF image, the non-CAA plaque type region is labeled as a diffuse plaque; otherwise, the region is labeled as a core plaque. The coordinates of each annotation data in the CARS / TPEF bimodal image are located in the β-amyloid plaque immunofluorescence image and the corresponding region is magnified. After verifying the annotation, the final bimodal annotation file is obtained.

4. The method according to claim 3, characterized in that, During the annotation process of CARS and TPEF images, patches are selected by keyboard and mouse input to create a queryable dictionary (dict), and the spatial position and size of the labeled labels and marker boxes are stored in memory and saved as an XML file.

5. The method according to claim 3, characterized in that, The three-channel RGB image formed by merging the CARS image and TPEF image has the R channel as the CARS image, the G channel as the TPEF image, and the B channel as the TPEF image minus its own mean. In the CARS / TPEF bimodal images, labeled images are generated based on the label type and bounding box parameters in each labeled file and saved as XML format files, which record the spatial location and type of core, diffuse, and CAA type β-amyloid plaques.

6. The method according to claim 1, characterized in that, Before annotating the TPEF image, the process also includes: preprocessing the TPEF image using an image enhancement algorithm to enhance the features of patches with strong autofluorescence.

7. The method according to claim 1, characterized in that, The preset neural network is a YOLOv4 object recognition network model. The training parameters are: the number of iterations (epochs) is between 100 and 500, the optimizer uses the Adam optimization algorithm, and the loss function uses the full intersection-union loss and the binary cross-entropy loss, which are used to calculate the bounding box loss, confidence loss, and classification loss, respectively. The learning rate is between 0.001 and 0.0001, the exponential decay rate of the first moment estimation is 0.9, the exponential decay rate of the second moment estimation is 0.999, and the decay rate is 0.

8. A label-free β-amyloid plaque identification device, characterized in that, The device includes: The image acquisition module is used to acquire TPEF and CARS images of coronal sections of APP / PS1 mouse brains of a preset thickness. The TPEF image contains autofluorescence signals of endogenous fluorophores, and the CARS image contains spectral signals. The annotation file acquisition module is used to annotate different β-amyloid plaques on TPEF images and CARS images using preset annotation tools to obtain annotation files; The bimodal annotation file acquisition module is used to obtain annotated images based on the coordinates and types in the annotation file using CARS / TPEF bimodal images as masks. After traversing all the annotation data in the annotation file, the annotations are reviewed to obtain annotated images representing the locations of plaques of different pathological types. The target recognition network model acquisition module is used to divide the three-channel CARS / TPEF bimodal images and corresponding annotation files into training and validation sets according to a preset ratio, train a preset neural network based on the training set, and validate the trained model based on the validation set to obtain a model that meets the preset requirements as a bimodal target recognition network model. The plaque recognition module is used to input the CARS image and TPEF image to be identified into the bimodal target recognition network model, and output the spatial location, pathological type and corresponding confidence level of the β-amyloid plaque.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the label-free β-amyloid plaque identification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the label-free β-amyloid plaque identification method according to any one of claims 1 to 7.

Images