Method based on fluorescent slide multi-layer fusion and ai multi-layer fusion processing, and use thereof

Abstract

A method and apparatus based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing. The method comprises: determining the height of a camera, the thickness of tissue, and the depth of field of a lens, so as to calculate photographic parameters; by means of an AI model, extracting valid information of multiple image layers; and using feature fusion technology to generate an image where each layer is clear. The apparatus comprises a fluorescence scanning module, an image processing module, and an image fusion module. By means of the method and apparatus, the spatial resolution and multi-labeling accuracy of fluorescent microscopy images are improved.

Description

Method and application based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing TECHNICAL FIELD

[0001] The present application relates to the field of fluorescence microscopy imaging technology, in particular to a method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing and its application. BACKGROUND

[0002] Fluorescence microscopy is an important tool in biological research, which can mark specific structures through fluorescent markers, and thus realize high-resolution observation of internal structures of cells or tissue samples. In recent years, with the development of imaging technology and fluorescent labeling technology, fluorescent slide multi-layer fusion technology has been widely used as a method to enhance the depth and details of microscopic images. This technology usually involves stacking multiple slides with different fluorescent markers together and processing the images through computer software to obtain the final composite image.

[0003] However, in practical applications, due to the different thicknesses of tissue samples and the limitations of camera depth of field, traditional multi-layer fusion technology faces some challenges. In particular, when the thickness of the sample exceeds the depth of field of the camera, different levels of cells in the image will show different degrees of blur, which not only reduces the quality of the image, but also may lead to misidentification of cell features. In addition, due to the different optical properties of cells at different levels, how to effectively extract clear and effective cell information from these levels has become a problem to be solved.

[0004] Current technical means for solving the above problems are not mature, especially in how to ensure that cells at different levels can be clearly presented and how to accurately fuse effective image information at different levels, existing technical solutions often fail to achieve satisfactory results. Therefore, it is urgent to develop new technical means to overcome the shortcomings of existing technology. SUMMARY

[0005] The present application provides a method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing and its application, which solves the problem of immature technical solutions in the prior art.

[0006] The core technology of the present application is a method for improving the quality of multi-layer fluorescent slide microscopic images through automatic scanning, AI image processing and multi-layer fusion technology, which solves the problem of cell blur caused by tissue thickness exceeding the depth of field of the camera.

[0007] In a first aspect, the present application provides a method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing, which comprises the following steps:

[0008] S00, fluorescence scanning logic:

[0009] Determine the camera height H, the tissue thickness W and the lens depth of field f of the camera to calculate the number of pictures N, the offset S and the starting height C;

[0010] Move the Z axis of the camera to the starting height C, trigger a picture every offset S distance until N pictures are taken and move to the next field of view to repeat the shooting, and then get the multi-layer fluorescence microscope image data after the current slide is shot;

[0011] The multi-layer fluorescence microscope image data covers different depths and planes;

[0012] S10, AI image processing to obtain effective information:

[0013] Extract different levels of effective information of multi-layer image data through the AI fluorescence image multi-layer model, so as to extract the clear part of the image from each layer of image, and the AI fluorescence image multi-layer model is trained by collecting multi-layer fluorescence microscope image data of historical data;

[0014] S20, AI fluorescence image multi-layer fusion:

[0015] Extract the features of each image in the multi-layer fluorescence microscope image data and merge the features;

[0016] Superimpose or weighted sum the feature maps from different fluorescence channels to obtain the integrated comprehensive feature expression;

[0017] Use spatial pyramid pooling technology of different scales to summarize and integrate feature maps of different resolutions to capture multi-scale spatial information;

[0018] Combine feature representations from different levels, different scales or different fields of view to realize multi-scale feature fusion, improve the comprehensive understanding and description ability of complex structures of samples through fusion strategy, and then fuse all clear images together to generate an image with clear layers.

[0019] Further, the training step of the AI fluorescence image multi-layer model in S10 includes:

[0020] Collect historical multi-layer fluorescence microscope image data to ensure that the data set covers images of different depths and planes, including samples labeled with multiple fluorescence probes;

[0021] Preprocess the image data;

[0022] Select a deep learning model suitable for processing multi-level information and train the model using fluorescence slide image data with multi-layer structure;

[0023] The trained model is evaluated, the performance is tested using the validation dataset, and the model is optimized according to the evaluation results to improve the spatial resolution and the accuracy of multiple markers;

[0024] The trained model is applied to the actual fluorescence scanning logic to extract effective information at different levels.

[0025] Further, the convolutional neural network is selected, and the model is trained using fluorescence slide image data with a multi-layer structure.

[0026] Further, the preprocessing includes contrast enhancement and normalization.

[0027] Further, in the S20 step, after extracting the features of each image in the multi-layer fluorescence microscope image data, wavelet transform or convolutional neural network is used for feature merging.

[0028] Further, in the S20 step, the information of different fluorescence probe markers is comprehensively analyzed through the integrated comprehensive feature expression.

[0029] Further, the calculation formula of the number of photographs N, the offset S, and the starting height C is:

[0030] N=W / (f / 2);

[0031] S=H / N;

[0032] C=H-N / 2*S.

[0033] In a second aspect, the present application provides a fluorescence slide multi-layer fusion process and AI multi-layer fusion processing device, comprising:

[0034] The fluorescence scanning module is used to input the camera height H, the tissue thickness W, and the lens depth of field f of the clear field of view, to calculate the number of photographs N, the offset S, and the starting height C; the Z-axis of the camera is moved to the starting height C, and the camera is triggered once every offset S, until N pictures are taken and moved to the next field of view for repeated shooting, and the multi-layer fluorescence microscope image data is obtained after the current slide is shot.

[0035] The multi-layer fluorescence microscope image data covers different depths and planes; the calculation formula of the number of photographs N, the offset S, and the starting height C is:

[0036] N=W / (f / 2); S=H / N; C=H-N / 2*S;

[0037] The AI image processing module extracts different levels of effective information from multi-layer image data through an AI fluorescent image multi-layer model, thereby extracting clear images from each layer of image.

[0038] The AI fluorescent image multi-layer fusion module extracts features of each image in the multi-layer fluorescent microscope image data and performs feature merging.

[0039] The feature maps from different fluorescence channels are superimposed or weighted summed to obtain integrated comprehensive feature expression.

[0040] Multi-scale convolution kernels and pooling operations are used to extract features from multiple levels, and then these features are cascaded or processed in parallel to enhance the understanding ability of cell structure and tissue distribution.

[0041] Multi-scale feature fusion is achieved by combining feature representations from different levels, different scales or different fields of view, and the overall understanding and description ability of complex sample structures is improved through fusion strategies, so that all clear images are fused together to generate an image that is clear at each level.

[0042] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the method of multi-layer fusion process based on fluorescent slides and AI multi-layer fusion processing described above.

[0043] In a fourth aspect, the present application provides a readable storage medium, the readable storage medium storing a computer program, the computer program comprising program code for controlling a process to perform the process, the process comprising the method of multi-layer fusion process based on fluorescent slides and AI multi-layer fusion processing described above.

[0044] The main contributions and innovations of the present application are as follows:

[0045] 1. The problem of blurred cells at different levels due to tissue thickness exceeding the camera depth of field is solved: the appropriate number of photographs, offset and starting height are determined by automatic scanning logic to ensure that each level of cells can be clearly imaged.

[0046] 2. Effectively extract effective information at different levels: use AI image processing technology to extract clear parts from each layer of image, improve the usability and analysis value of the image.

[0047] 3. Improve the quality of image fusion: through feature level fusion, channel level fusion, spatial pyramid pooling, attention mechanism and other fusion strategies, effectively integrate information at different levels, and obtain clearer and more accurate cell and tissue structure images.

[0048] 4. Enhanced understanding of complex sample structures: Utilizing multi-scale convolutional kernels and pooling operations, the understanding of cellular structures and tissue distribution is enhanced, contributing to a more comprehensive understanding of the complex structure of the sample.

[0049] 5. Improved efficiency and accuracy: Automated processing reduces the need for manual operations, improving processing speed and the accuracy of image analysis.

[0050] 6. Optimized spatial resolution and accuracy of multiple markers: Through the training and optimization of AI models, the spatial resolution and accuracy of multiple markers are improved, contributing to more accurate identification and analysis of cellular features.

[0051] 7. Improved overall efficiency of image processing: Through integrated fluorescence scanning, image processing, and fusion techniques, an efficient image processing workflow is achieved, accelerating research and diagnosis.

[0052] In summary, the present invention significantly improves the quality and efficiency of multi-layer fluorescence slide microscopic images through automated scanning logic, advanced AI image processing technology, and multi-layer fusion methods, providing strong support for biological research, medical diagnosis, and other fields.

[0053] The details of one or more embodiments of the present invention are presented in the following drawings and description, so that other features, objects, and advantages of the present invention are more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0054] The drawings described herein are intended to provide a further understanding of the present invention, and form a part of the present invention. The illustrative embodiments of the present invention and their description serve to explain the present invention. They are not intended to limit the present invention unduly.

[0055] Figure 1 is a flowchart of a method based on a multi-layer fusion process of a fluorescence slide and AI multi-layer fusion processing according to an embodiment of the present invention;

[0056] Figure 2 is a first layer image taken according to an embodiment of the present invention;

[0057] Figure 3 is a second layer image taken according to an embodiment of the present invention;

[0058] Figure 4 is a third layer image taken according to an embodiment of the present invention;

[0059] Figure 5 is a fourth layer image taken according to an embodiment of the present invention;

[0060] Figure 6 is a fifth layer image taken according to an embodiment of the present invention;

[0061] Figure 7 is a sixth layer image taken according to an embodiment of the present invention;

[0062] Fig. 8 is a first layer image extracted according to an embodiment of the present application;

[0063] Fig. 9 is a second layer image extracted according to an embodiment of the present application;

[0064] Fig. 10 is a third layer image extracted according to an embodiment of the present application;

[0065] Fig. 11 is a fourth layer image extracted according to an embodiment of the present application;

[0066] Fig. 12 is a fifth layer image extracted according to an embodiment of the present application;

[0067] Fig. 13 is a sixth layer image extracted according to an embodiment of the present application;

[0068] Fig. 14 is a fused image according to an embodiment of the present application;

[0069] Fig. 15 is a hardware structure diagram of an electronic device according to an embodiment of the present application. Embodiments of the present application

[0070] The exemplary embodiments will be described in detail herein below with reference to the drawings. In the following description, the same drawings reference numbers are used to denote like or similar elements. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with one or more embodiments of the present specification. Rather, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present specification as detailed in the appended claims.

[0071] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in the present specification in other embodiments. In some other embodiments, the steps included in the methods can be more or less than those described in the present specification. In addition, a single step described in the present specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the present specification can be combined into a single step for description in other embodiments.

[0072] Since the lens has a depth of field, the image is clear within a certain height range, but the product on the fluorescent slide itself also has a thickness. When the thickness of the product is greater than the depth of field of the lens, the image will have a local area that appears blurred. Therefore, due to the different thicknesses of the product, the depth of field of the camera cannot cover the different levels of cells, and the partial layer cells appear blurred and unclear, and the traditional multi-layer fusion has recognition errors, resulting in a deviation between the fused image and the actual image.

[0073] Based on this, the present application solves the problems existing in the prior art based on the multi-layer fusion process of fluorescent slides and AI multi-layer fusion processing.

[0074] Embodiment one

[0075] The present application aims to propose a method based on the multi-layer fusion process of fluorescent slides and AI multi-layer fusion processing, which significantly improves the quality and analysis efficiency of multi-layer fluorescent slide microscopic images through automated scanning logic, advanced AI image processing technology and multi-layer fusion method, providing strong support for biological research, medical diagnosis and other fields.

[0076] Specifically, the present application provides a method based on the multi-layer fusion of fluorescent slides and AI multi-layer fusion processing, specifically, referring to FIG. 1, the method comprises:

[0077] 1. Scanning process

[0078] 1.1. Offset calculation S, number of pictures calculation N starting height C: input the height H of clear field of view, input the thickness W of tissue, input the lens depth of field f, the calculation formula is as follows:

[0079] N=W / (f / 2), S=H / N, C=H-N / 2*S.

[0080] 1.2. Different layer photographing: move the Z axis to the starting height C, trigger the photographing every S distance, and move to the next field of view after taking N pictures.

[0081] As shown in FIGS. 2-7, they are images of the first layer to the sixth layer, respectively.

[0082] 2. AI image processing to obtain effective information: obtain effective information in the image through AI image processing. The training steps of the AI fluorescent image multi-layer training model are as follows:

[0083] 2.1 Data collection and preparation: collect high-quality multi-layer fluorescent microscope image data, covering different depths and planes. Ensure that the image data contains samples labeled with multiple fluorescent probes, so that the model learns multiple marker information.

[0084] 2.2 Data preprocessing: preprocess the image data, enhance contrast, normalize, etc. to optimize the training effect. The preprocessing here is the general operation of model training, and the specific steps and principles will not be repeated here.

[0085] 2.3 Model selection and training: select a deep learning model convolutional neural network (CNN) for processing multi-layer information. Use fluorescent slide image data with multi-layer structure to train the model to learn cell structures or tissue distribution at different depths and planes.

[0086] 2.4 Evaluate the trained model and test its performance using the validation dataset. Based on the evaluation results, optimize the model to improve spatial resolution and accuracy of multiple markers

[0087] 2.5 Apply the trained model to extract effective information at different levels in the actual fluorescence scanning logic.

[0088] That is, after each field of view is photographed, use the trained model to extract, after extracting all layer information, switch to the next field of view for shooting, and after shooting, call the model again for extraction, and so on.

[0089] 3. Fluorescence image multi-level fusion method

[0090] 3.1 Feature-level fusion: Extract features from each image using wavelet transform or convolutional neural network (CNN) technology, then merge these features. For example:

[0091] 3.1.1 Feature extraction

[0092] Use convolutional neural network (CNN) or other appropriate feature extraction algorithms to extract features from each image. CNN can extract local and global features in images through a series of convolutional layers, activation functions, pooling layers, etc.

[0093] 3.1.2 Feature merging

[0094] Wavelet transform:

[0095] Apply wavelet transform to multi-resolution analysis of images to obtain feature representations at different scales. Select key coefficients from the wavelet transform results as feature representations. Merge wavelet coefficients of different layers, which can be simple weighted average, maximum selection or other strategies.

[0096] Convolutional neural network (CNN):

[0097] Use the output of the intermediate layers of CNN as feature representations. You can concatenate or splice CNN feature maps of different layers. You can also merge features through fully connected layers or other fusion layers.

[0098] 3.1.3 Fusion strategy

[0099] Choose the appropriate fusion strategy according to the specific situation:

[0100] Weighted average: Assign different weights to the features of each layer and sum them up.

[0101] Maximum selection: Select the maximum value at each position.

[0102] Cascade / Concatenation: Concatenate feature maps from different layers to form a larger feature vector.

[0103] Attention Mechanism: Dynamically adjust the importance of different features through attention weights, achieving intelligent fusion.

[0104] 3.1.4 Post-processing

[0105] Further processing of the fused features, such as normalization, dimensionality reduction, etc., to improve the efficiency and effectiveness of subsequent processing.

[0106] Through the above steps, features can be effectively extracted from different layers of images, and wavelet transform or convolutional neural network can be used for feature merging to achieve feature-level fusion. This fusion method can better preserve the details of different layers and improve the quality and information content of the final image.

[0107] 3.2 Channel-level fusion: Superimpose or weighted sum the feature maps from different fluorescence channels to obtain integrated comprehensive feature expression. Comprehensive analysis of information labeled by different fluorescent probes. For example:

[0108] 3.2.1 Feature map superposition

[0109] For feature maps from different fluorescence channels, pixel-level superposition can be directly performed. Assuming there are two feature maps F1 and F2, they have the same size H*W*C (where H and W are height and width, and C is the number of channels), directly add the two feature maps to obtain a new feature map F stacked :

[0110]

[0111] where x and y are pixel coordinates, and c is the channel index.

[0112] 3.2.2 Weighted sum

[0113] If the importance of different channels needs to be considered, a weighted sum method can be used. Assuming each feature map F i has a weight w i , then the comprehensive feature map F weighted is:

[0114]

[0115] where n is the number of feature maps.

[0116] 3.2.3 Weight calculation

[0117] The weight w i can be determined in several ways:

[0118] Manual assignment: manually assign weights to each channel based on experience or prior knowledge.

[0119] Attention mechanism: dynamically adjust the weight of each channel by training an attention model. The attention model can be another small CNN or a simple fully connected layer that dynamically assigns weights based on the content of the feature map.

[0120] Adaptive weights: use adaptive weight methods, such as automatically calculating weights based on the correlation or difference between feature maps.

[0121] 3.2.4 Post-processing

[0122] Further processing of the fused feature map, such as normalization, dimensionality reduction, etc., to improve the efficiency and effectiveness of subsequent processing.

[0123] Through the above steps, features can be effectively extracted from images of different fluorescence channels, and feature maps can be superimposed or weighted summed to realize channel-level fusion. This fusion method can better preserve the information marked by different fluorescent probes and improve the quality and information content of the final image.

[0124] 3.3 Spatial pyramid pooling: use different scale spatial pyramid pooling techniques to aggregate and integrate feature maps of different resolutions, capturing multi-scale spatial information. For example:

[0125] 3.3.1 Pyramid level division

[0126] Divide the extracted feature map into multiple different scale levels to form a pyramid structure. Each pyramid level corresponds to a different pooling window size, such as using 1X1, 2X2, 4X4, etc. Different size pooling windows.

[0127] 3.3.2 Pooling operation

[0128] Perform pooling operation on the feature map of each pyramid level. Pooling operation can be max pooling or average pooling, used to reduce the resolution of feature map while maintaining the invariance of features. The result of each pooling operation will produce a fixed size feature vector.

[0129] 3.3.3 Feature merging

[0130] Merge the feature vectors produced by different levels of pooling operation to form a unified fixed size feature vector. Feature vector merging can be done by simple splicing or cascading.

[0131] 3.3.4 Post-processing

[0132] The combined feature vectors are further processed, such as normalization, dimension reduction, etc., to improve the efficiency and effect of subsequent processing.

[0133] Through the above steps, multi-scale spatial information can be effectively extracted from feature maps of different resolutions, and summarized and integrated through spatial pyramid pooling technology. This fusion method can better preserve information of different scales and improve the quality and information content of the final image.

[0134] 3.4 Multi-scale feature fusion: Combine feature representations from different levels, different scales, or different fields of view to improve the comprehensive understanding and description of the complex structure of the sample through appropriate fusion strategies.

[0135] This process emphasizes the use of different receptive fields to capture various types of information in images. Different levels refer to different layers in the network; different scales generally refer to using different sizes of convolution kernels to process input data; different fields of view usually refer to observing data from different angles or positions.

[0136] Thus, the overall workflow is as follows: the starting height C of the Z-axis movement is obtained, a camera is triggered every S distance to obtain an image corresponding to the current height, such as Fig. 2 (first layer) to Fig. 7 (sixth layer), the clear part of the image is extracted in the current image, such as Fig. 8 (first layer) to Fig. 13 (sixth layer), one cycle of N images, after all the images are taken, the clear part of all the images is fused together to finally generate a synthesized image with clear images of each layer, such as Fig. 14 (fused image), which covers images of different heights, making it display and obtain more effective information.

[0137] It can be seen that the present application realizes digital pathology fluorescence multi-layer processing, especially a new processing flow and logic of a digital pathology fluorescence device, uses a deep learning model, a convolutional neural network (CNN), which can extract high-level features from each image. These features can capture more complex image patterns, such as texture, shape, and semantic information. The AI model can learn the relationships and trade-offs between different images, automatically adjusting weights or operations during the fusion process to achieve the best fusion results. This adaptability can improve the quality and accuracy of the fusion results. AI can be applied to pixel-level operations such as denoising, super-resolution, and color correction to improve the visual quality and information clarity of the images. Efficient deep learning models can process large amounts of image data in a short time, suitable for applications that require real-time processing and response. Using AI-based large models for image fusion not only improves the quality and effect of fusion, but also enables automation and efficient processing, suitable for a variety of complex application requirements.

[0138] Example Two

[0139] Based on the same concept, the application also provides a multi-layer fusion process and AI multi-layer fusion processing device based on a fluorescence slide, comprising:

[0140] A fluorescence scanning module is configured to input the camera height H, the tissue thickness W, and the lens depth of field f of the camera, calculate the number of pictures N, the offset S, and the starting height C, move the Z-axis of the camera to the starting height C, trigger a picture taking every offset S, and move to the next field of view to repeat the picture taking after N pictures are taken, and obtain multi-layer fluorescence microscope image data after the picture taking of the current slide is completed.

[0141] The multi-layer fluorescence microscope image data covers different depths and planes, and the calculation formula of the number of pictures N, the offset S, and the starting height C is:

[0142] N = W / (f / 2); S = H / N; C = H-N / 2*S;

[0143] An AI image processing module extracts different levels of effective information of the multi-layer image data through an AI fluorescence image multi-layer model, thereby extracting the clear part of the image from each layer of the image, and the AI fluorescence image multi-layer model is trained through the collection of multi-layer fluorescence microscope image data of historical data.

[0144] An AI fluorescence image multi-layer fusion module extracts the features of each image in the multi-layer fluorescence microscope image data and merges the features.

[0145] The feature maps from different fluorescence channels are superimposed or weighted summed to obtain the integrated comprehensive feature expression.

[0146] A spatial pyramid pooling technology of different scales is used to collect and integrate feature maps of different resolutions, thereby capturing multi-scale spatial information.

[0147] Based on the attention mechanism, the feature level fusion dynamically adjusts the weight of the feature according to the importance of different channels or spatial positions, so as to enhance the representation ability of specific information.

[0148] Multi-scale convolution kernels and pooling operations are used to extract features from multiple levels, and then these features are cascaded or processed in parallel to enhance the understanding ability of cell structure and tissue distribution.

[0149] Multi-scale feature fusion is realized by combining feature representations from different levels, different scales, or different fields of view, and the overall understanding and description ability of complex structures of samples is improved through the fusion strategy, so that all clear part images are fused together to generate an image with clear layers.

[0150] Embodiment three

[0151] The embodiment also provides an electronic device, referring to FIG. 15, comprising a memory 404 and a processor 402, the memory 404 storing a computer program, and the processor 402 being configured to run the computer program to perform the steps in any of the above method embodiments.

[0152] Specifically, the processor 402 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0153] The memory 404 can include a mass storage that stores data or instructions. For example, and without limitation, the memory 404 can include a Hard Disk Drive (HDD), a floppy disk drive, a Solid State Drive (SSD), a flash drive, a Compact Disc Read Only Memory (CD-ROM), a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 404 can be removable and / or non-removable (or fixed) as appropriate. The memory 404 can be internal or external as appropriate. In particular embodiments, the memory 404 is a Non-Volatile memory. In particular embodiments, the memory 404 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH memory, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random Access Memory (FPMDRAM), an Extended Data Output Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0154] The memory 404 can be used to store or buffer various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 402.

[0155] The processor 402 implements any of the above-mentioned methods of fluorescence slide multi-layer fusion and AI multi-layer fusion processing by reading and executing the computer program instructions stored in the memory 404.

[0156] Optionally, the above-mentioned electronic device can further include a transmission device 406 connected with the processor 402 and an input / output device 408 connected with the processor 402.

[0157] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network can include wired or wireless networks provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (NIC) which can be connected with other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0158] The input / output device 408 is used to input or output information.

[0159] Embodiment Four

[0160] The embodiment also provides a readable storage medium, and the readable storage medium stores a computer program. The computer program includes program codes for controlling a process to execute the process. The process includes the method of fluorescence slide multi-layer fusion and AI multi-layer fusion processing according to the embodiment one.

[0161] It should be noted that the specific examples in the embodiment can refer to the examples described in the above-mentioned embodiments and optional implementation manners, and the embodiment will not be described here.

[0162] Generally, various embodiments can be implemented in hardware or special-purpose circuitry, software, logic or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, microprocessor or other computing device, but the application is not limited thereto. Although various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0163] Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, can be stored in any apparatus-readable data storage medium and they include program instructions to implement certain tasks. The program product can include one or more computer-executable components tangibly embodied in a computer- readable medium, when executed, for implementing one or more embodiments of the present application. The one or more computer-executable components can be one or more of: a process; a function; a routine; a sag; a subroutine; a plug-in; an app; or a program. The one or more computer-executable components can include a computer program that is written in any form of programming language, including code.

[0164] Those skilled in the art should clearly understand that each technical feature in the above embodiments can be combined with any other technical feature, and for the sake of brevity, each technical feature in the above embodiments has not been described in all possible combinations, however, as long as the combinations of the technical features do not exist contradictions, it should be considered that they are within the scope of the present disclosure.

[0165] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing, characterized in that, Comprising the following steps: S00, fluorescence scanning logic: Determine the camera height H, tissue thickness W and the lens depth of field f of the camera to calculate the number of pictures N, the offset S and the starting height C; Move the Z axis of the camera to the starting height C, trigger a picture every offset S distance until N pictures are taken, then move to the next field of view and repeat the shooting, and finally get the multi-layer fluorescence microscope image data after shooting the current slide; Wherein, the multi-layer fluorescence microscope image data covers different depths and planes; S10, AI image processing to obtain effective information: Extract different levels of effective information from multi-layer image data through AI fluorescence image multi-layer model, so as to extract the clear part of the image from each layer of image, and the AI fluorescence image multi-layer model is trained by collecting historical multi-layer fluorescence microscope image data; S20, AI fluorescence image multi-layer fusion: Extract the features of each image in the multi-layer fluorescence microscope image data and merge the features; Superimpose or weighted sum the feature maps from different fluorescence channels to obtain the integrated comprehensive feature expression; Use spatial pyramid pooling technology of different scales to summarize and integrate feature maps of different resolutions to capture multi-scale spatial information; Combine feature representations from different levels, different scales or different fields of view to realize multi-scale feature fusion, improve the comprehensive understanding and description ability of complex structures of samples through fusion strategy, and then fuse all clear images together to generate an image with clear layers.

2. The method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing according to claim 1, characterized in that, In step S10, the training steps of the AI fluorescence image multi-layer model include: Collect historical multi-layer fluorescence microscope image data to ensure that the data set covers images of different depths and planes, including samples labeled with multiple fluorescence probes; Preprocess the image data; Select a deep learning model suitable for processing multi-level information and train the model using fluorescence slide image data with multi-layer structure; Evaluate the trained model, test its performance using a validation data set, and optimize the model according to the evaluation results to improve spatial resolution and multi-label accuracy; Apply the trained model to the actual fluorescence scanning logic to extract different levels of effective information.

3. The method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing according to claim 2, characterized in that, Select a convolutional neural network and train the model using fluorescence slide image data with multi-layer structure.

4. The method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing according to claim 2, wherein, Preprocessing includes contrast enhancement and normalization.

5. The method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing according to claim 1, characterized in that, In step S20, after extracting the features of each image in the multi-layer fluorescence microscope image data, use wavelet transform or convolutional neural network for feature merging.

6. The method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing according to claim 1, wherein, In step S20, integrate the information of different fluorescence probes through the integrated comprehensive feature expression.

7. The method based on fluorescent slide multi-layer fusion and AI multi-layer fusion processing according to any one of claims 1-6, characterized in that, The calculation formula of the number of pictures N, the offset S and the starting height C is: N = W / (f / 2); S = H / N; C = H-N / 2*S.

8. A fluorescence slide multi-layer fusion process and AI multi-layer fusion processing device based on, characterized in that, Comprising: The fluorescence scanning module is used to input the camera height H, the tissue thickness W and the lens depth of field f of the camera, so as to calculate the number of pictures N, the offset S and the starting height C; the Z axis of the camera is moved to the starting height C, and the camera is triggered once every offset S until N pictures are taken, and then the camera is moved to the next field of view for repeated shooting, and the multi-layer fluorescence microscope image data of the current slide is obtained after the shooting; The multi-layer fluorescence microscope image data covers different depths and planes; the calculation formula of the number of pictures N, the offset S and the starting height C is: N = W / (f / 2); S = H / N; C = H-N / 2*S; The AI image processing module extracts different levels of effective information of the multi-layer image data through the AI fluorescence image multi-layer model, so as to extract the clear part of the image from each layer of the image, and the AI fluorescence image multi-layer model is trained through the multi-layer fluorescence microscope image data of the historical data collected; The AI fluorescence image multi-layer fusion module extracts the features of each image in the multi-layer fluorescence microscope image data and merges the features; The feature maps from different fluorescence channels are superimposed or weighted summed to obtain the integrated comprehensive feature expression; The spatial pyramid pooling technology of different scales is adopted to collect and integrate the feature maps of different resolutions, so as to capture multi-scale spatial information; Multi-scale feature fusion is realized by combining feature representations from different levels, different scales or different fields of view, the comprehensive understanding and description ability of complex structures of samples is improved through the fusion strategy, so that all the clear part of the image is fused together to generate an image with clear layers. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to run the computer program to execute the method of claim 1 to 7 based on the fluorescence slide multi-layer fusion process and the AI multi-layer fusion processing.

10. A readable storage medium, characterized by, The readable storage medium stores a computer program, and the computer program includes program code for controlling a process to execute the process, and the process includes the method of claim 1 to 7 based on the fluorescence slide multi-layer fusion process and the AI multi-layer fusion processing.

Images