Dual-optical-path image detection and rendering method and device

Abstract

The invention relates to the technical field of image processing, in particular to a dual-optical-path image detection and rendering method and device. The method comprises the following steps: constructing a multi-thread processing flow; acquiring a bright light field image and a fluorescent light field image of the cell; based on a multi-thread processing flow, carrying out image calibration, bright light field image detection and fluorescent field image detection on the bright light field image and the fluorescent field image of the cell at the same time to obtain a calibration image, a bright light field image detection result and a fluorescent field image detection result; performing image alignment based on the detection result images of the bright light field image and the fluorescent light field image to obtain an aligned image; and performing fluorescence intensity estimation and color rendering on the aligned image to complete dual-light-path image detection and rendering. The invention aims to solve the registration problem caused by imaging condition difference between the fluorescent field image and the bright light field image, and provides an effective method to estimate the fluorescence intensity of the cells in the fluorescent field image and accurately render the fluorescence intensity to the corresponding cell position in the bright light field image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a dual-light path image detection and rendering method and device. Background Art

[0002] Overview of single-cell analysis technology: Single-cell analysis refers to various biological tests and analyses performed at the level of a single cell. With the development of biotechnology, single-cell analysis has become an important means to study cell heterogeneity, intercellular communication, cell function, and the mechanism of disease occurrence. Single-cell analysis technology covers a variety of methods, including but not limited to single-cell sequencing, single-cell proteomics, single-cell metabolomics, etc. These technologies enable researchers to obtain gene expression patterns, protein composition, and other molecular information of single cells, thereby gaining a deep understanding of cell behavior and function. In recent years, single-cell analysis technology has developed rapidly, especially in single-cell transcriptomics, which has achieved a leap from whole transcriptome sequencing to spatial transcriptomics. These technologies not only help scientists identify different subpopulations within cells, but also reveal the dynamic changes of these subpopulations in time and space, providing unprecedented resolution for studying complex biological systems. For example, in cancer research, single-cell analysis technology can distinguish between tumor cells and normal cells, and can even be further refined to different subtypes of tumor cells, which is of great significance for the design of precision medicine and personalized treatment plans.

[0003] Importance of cell fluorescence intensity: In single-cell analysis, fluorescence labeling technology is widely used for visualization and quantitative analysis of intracellular molecules. By introducing fluorescent proteins (such as GFP, RFP, etc.) or other fluorescent dyes, researchers can label target molecules at specific locations in cells or throughout the cell. Fluorescence intensity reflects the number or activity state of the labeled molecules. This technology is particularly suitable for tracking changes in intracellular molecules, such as protein expression levels, enzyme activity, cell cycle stages, etc. Accurate measurement of fluorescence intensity is essential for understanding the state of cells and how they respond to external stimuli. Fluorescence labeling technology is not limited to static image acquisition, but can also be combined with live cell imaging technology to achieve real-time observation of dynamic processes in cells. For example, through fluorescence resonance energy transfer (FRET) technology, scientists can observe the interactions between proteins in cells; using fluorescence lifetime imaging microscopy (FLIM), the microenvironmental changes of intracellular molecules can be further analyzed. Fluorescence intensity, as a quantitative indicator, plays an indispensable role in single-cell analysis. It not only provides qualitative visual information, but more importantly, it can be quantitatively analyzed, providing a basis for subsequent data interpretation and functional verification.

[0004] Image calibration technology: Image calibration becomes particularly important when processing images from different light sources, especially when fusing fluorescence field images and bright field images. Image calibration refers to the process of aligning and standardizing images obtained under different imaging conditions to ensure comparability between two or more images. This process usually involves steps such as geometric correction, color correction, brightness and contrast adjustment, with the aim of eliminating the effects of differences in imaging equipment and conditions and ensuring the reliability and consistency of image data. Image calibration technology is particularly critical in multimodal imaging because different imaging techniques often have different imaging characteristics. For example, fluorescence imaging usually has high specificity and sensitivity, but is easily interfered by background fluorescence; while bright field imaging can provide good anatomical information, but lacks functional information at the molecular level. Through precise image calibration, the advantages of the two imaging modes can be complemented to obtain more comprehensive cell information. In addition, image calibration is also a prerequisite for applications such as cell positioning and cell morphology analysis.

[0005] The main difficulty of the existing technology is how to calibrate the bright field image and the fluorescent field image, so as to achieve the alignment of the bright field cells and the fluorescent field cells. The current image calibration technologies mainly include:

[0006] ① Calibration based on feature points: This type of method first detects stable feature points (such as corners, edges, textures, etc.) in the two images, then calculates the correspondence between these feature points, and finally calculates the geometric transformation parameters through these correspondences to achieve the registration of the two images. This method is relatively mature and applicable to a variety of image types, especially when there are obvious feature points in the image.

[0007] ② Calibration based on global transformation: This method is simple, intuitive and easy to implement. It usually assumes that there is some global transformation (such as translation, rotation, scaling, etc.) between the two images, and solves the optimal transformation parameters by minimizing some error function. Common methods include affine transformation, similarity transformation, etc.

[0008] ③ Calibration based on mutual information: This type of method does not require obvious feature points between images and is suitable for image registration of different modalities. It mainly seeks the best registration parameters by maximizing the mutual information between two images. Mutual information measures the degree of dependence between two images and is suitable for image registration with similar grayscale histograms.

[0009] ④ Calibration based on deep learning: The calibration method based on deep learning designs a convolutional neural network and trains the network to learn the correspondence between two images. This method usually includes steps such as generating training data sets, designing network architecture, training models, and testing models. It can learn complex nonlinear correspondences and is suitable for a variety of image types. Summary of the invention

[0010] In order to solve the technical problem of registering the fluorescence field image and the bright field image due to the difference in imaging conditions in the prior art, an embodiment of the present invention provides a dual-light path image detection and rendering method and device. The technical solution is as follows:

[0011] On the one hand, a dual-light path image detection and rendering method is provided, characterized in that the method comprises:

[0012] S1. Build a multi-threaded processing flow; obtain bright field images and fluorescent field images of cells;

[0013] S2. Based on a multi-threaded processing flow, image calibration, bright light field image detection and fluorescence field image detection are performed on the bright light field image and the fluorescence field image of the cell at the same time to obtain a calibration image, a bright light field image detection result and a fluorescence field image detection result;

[0014] S3, performing image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image;

[0015] S4. Perform fluorescence intensity estimation and color rendering on the aligned image to complete dual-light path image detection and rendering.

[0016] Optionally, in S2, image calibration is performed on the bright field image and the fluorescent field image of the cell, including:

[0017] Acquire a bright light field image and a fluorescent field image of the cell, and pre-process the bright light field image and the fluorescent field image;

[0018] A cross-anchor point alignment method is constructed, and image calibration is performed on the two preprocessed images based on the cross-anchor point alignment method to obtain a calibrated image.

[0019] Optionally, preprocessing the bright field image and the fluorescent field image includes:

[0020] Acquire bright field images and fluorescent field images of cells;

[0021] Convert the bright field image of the cell and the fluorescent field image of the cell into a grayscale image;

[0022] Dynamic threshold image enhancement is performed using the adaptive threshold method according to the characteristics of the image, and the image pixels are inverted.

[0023] Optionally, constructing a cross anchor point alignment method, performing image calibration on the two preprocessed images based on the cross anchor point alignment method to obtain a calibrated image, including:

[0024] Obtain the cell's bright light field preprocessed image and the cell's fluorescence field preprocessed image, use the probabilistic Hough transform algorithm, randomly select edge point pairs for line segment estimation, and extract the funnel-shaped area in the two images;

[0025] Detect the two straight lines on both sides of the funnel shape, extend the two straight lines, and find the intersection of the two extended lines;

[0026] According to the intersection point, the bright light field image and the fluorescent field image are calibrated to obtain a bright light field calibration image and a fluorescent field calibration image, and to obtain an offset between the bright light field image and the fluorescent field image.

[0027] Optionally, in S2, the fluorescence field image detection includes:

[0028] The fluorescence field image is converted into a grayscale image, and is screened according to a preset threshold to remove noise with a grayscale value less than the threshold;

[0029] Convert the filtered image into a binary image;

[0030] A connected domain focusing method was constructed to calculate the 8-connected domain of the target as the coarse detection result of the fluorescent field cells;

[0031] Small connected domains with close center points are merged into large connected domains to obtain the cell coordinates of the fluorescence field.

[0032] Optionally, in S2, bright field image detection includes:

[0033] The bright light field image is subjected to YOLO detection to obtain the bright light field cell coordinates.

[0034] Optionally, in S3, performing image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image includes:

[0035] Obtaining the offset between the bright field image and the fluorescence field image, as well as the fluorescence field cell coordinates and the bright field cell coordinates;

[0036] According to the offset of the bright light field image and the fluorescent field image, as well as the fluorescent field cell coordinates and the bright light field cell coordinates, the detection result images of the bright light field image and the fluorescent field image are aligned to obtain an aligned image.

[0037] Optionally, in S4, fluorescence intensity estimation and color rendering are performed on the aligned image to complete dual light path image detection and rendering, including:

[0038] The grayscale value is estimated based on the detection area of ​​the fluorescent field cell and mapped to the fluorescence intensity value range according to the following formula (1):

[0039] (1)

[0040] in, is the fluorescence intensity calculation coefficient, and G is the gray value of the fluorescent cell area.

[0041] According to the fluorescence intensity, Gaussian spots of different sizes are generated and rendered onto the bright light field cells as the fluorescence intensity visualization results of the bright light field cells, completing the dual-light path image detection and rendering of the cells.

[0042] On the other hand, a dual-light path image detection and rendering device is provided, which is applied to a dual-light path image detection and rendering method, and the device includes:

[0043] An image calibration module is used to obtain a bright field image of a cell and a fluorescent field image of a cell, and to preprocess the bright field image and the fluorescent field image; a cross anchor point alignment method is constructed, and image calibration is performed on the two preprocessed images based on the cross anchor point alignment method to obtain a calibrated image;

[0044] A fluorescent cell detection module is used to construct a connected domain focusing method, and to perform fluorescence detection on the bright field image and the fluorescent field image simultaneously based on the connected domain focusing method to obtain a detection result graph;

[0045] An image alignment module, used to perform image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image;

[0046] The fluorescence intensity rendering module is used to estimate the fluorescence intensity and perform color rendering on the aligned images, completing dual-light path image detection and rendering.

[0047] On the other hand, a dual-light path image detection and rendering device is provided, and the dual-light path image detection and rendering device includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the above-mentioned dual-light path image detection and rendering methods is implemented.

[0048] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dual-light path image detection and rendering methods.

[0049] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0050] In order to solve the problem that bright light field images and fluorescent field images from different light sources are difficult to align, an embodiment of the present invention proposes a cross-anchor point alignment method, which is not only simple and fast, but also can accurately align bright light field and fluorescent field images, is less affected by illumination changes, and does not require a large amount of labeled data for training.

[0051] Aiming at the problem of cell detection in bright field images and fluorescence field images, a connected domain focusing method is proposed, which can detect small-area cells in fluorescence field images and filter out small light spots caused by noise, and can accurately detect fluorescent cells in fluorescence field images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 is a flow chart of a dual-light-path image detection and rendering method provided by an embodiment of the present invention;

[0054] Figure 2 is a sample diagram of a bright field image and a fluorescent field image provided by an embodiment of the present invention;

[0055] Figure 3 is a preprocessing diagram of a bright field image and a fluorescent field image provided by an embodiment of the present invention;

[0056] Figure 4 It is a pixel map of an inverted image of dual-light-path image detection provided by an embodiment of the present invention;

[0057] Figure 5 It is a straight line detection diagram of dual light path image detection provided by an embodiment of the present invention;

[0058] Figure 6 The embodiment of the present invention provides a dual-light path image detection extending straight line to find the intersection diagram;

[0059] Figure 7 is a dual-light-path image calibration diagram provided by an embodiment of the present invention;

[0060] Figure 8 is a visualization diagram of the fluorescence field cell detection result provided by an embodiment of the present invention;

[0061] Fig. 9 It is a fluorescence intensity rendering visualization diagram provided by an embodiment of the present invention;

[0062] Fig.10 is a block diagram of a dual-light-path image detection and rendering device provided by an embodiment of the present invention;

[0063] Fig.11 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0065] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0066] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0067] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0068] The embodiment of the present invention provides a dual-light path image detection and rendering method, which can be implemented by a dual-light path image detection and rendering device, which can be a terminal or a server. Figure 1 The process flow of the dual-light path image detection and rendering method shown in the figure may include the following steps:

[0069] S1. Build a multi-threaded processing flow; obtain bright field images and fluorescence field images of cells.

[0070] In a feasible implementation manner, the bright field image and the fluorescent field image in the application scenario targeted by the present invention are as follows: Figure 2 As shown in the figure, the bright field image has rich structural information, while the fluorescent field image only has black and white structural information. For the calibration algorithm based on feature points, the feature points in the two images may not correspond or be difficult to identify, which may easily lead to feature point matching failure, that is, the calibration algorithm based on feature points is not suitable for this application scenario. For the calibration algorithm based on global transformation, there may be local deformation or non-rigid deformation between the bright field image and the fluorescent field image, and a simple global transformation may not be sufficient to accurately describe these deformations, resulting in poor calibration results. For the calibration algorithm based on mutual information, there are significant differences in the grayscale histograms of the bright field image and the fluorescent field image, and the mutual information may not be sufficient to capture the correspondence between the two images. For the calibration algorithm based on deep learning, it is difficult to generate high-quality training data sets using bright field images and fluorescent field images because there are large imaging differences between the two types of images. In addition, the training of deep learning models requires a lot of computing resources and time, which is not practical for real-time applications.

[0071] S2. Based on a multi-threaded processing flow, image calibration, bright light field image detection and fluorescence field image detection are performed on the bright light field image and the fluorescence field image of the cell at the same time to obtain a calibration image, a bright light field image detection result and a fluorescence field image detection result.

[0072] In a feasible implementation, Figure 1 As shown, the present invention first inputs 4 images, namely, bright field image B1 and fluorescence field image F1 for image calibration, and bright field image B2 and fluorescence field image F2 for cell detection. Then, three lines of image calibration, fluorescence field cell detection, and bright field cell detection are simultaneously run in a multi-threaded manner.

[0073] In a feasible implementation mode, according to Figure 2 It can be seen that the two images provide different types of information: bright field images are usually used to observe the overall morphology and structure of cells, while fluorescence field images are used to mark and quantify specific cell components or structures. Some of the main difficulties in the registration of bright field images and fluorescence field images are:

[0074] ① Differences in imaging conditions: Bright field images and fluorescence field images are usually obtained under different imaging conditions, which can lead to significant differences in resolution, brightness, contrast, and color between the images, thereby increasing the difficulty of registration.

[0075] ②Signal-to-noise ratio problem: Background noise may exist in fluorescence field images due to nonspecific binding, light scattering, or the properties of the fluorescent dye itself. High background noise will affect the accurate measurement of fluorescence intensity and may lead to incorrect selection of feature points during image registration. In contrast, bright field images may contain less noise, but their contrast and clarity may be affected by factors such as sample thickness and inconsistent refractive index.

[0076] ③ Feature selection and matching: In order to achieve image registration, it is necessary to find suitable feature points for matching. However, in bright field images and fluorescent field images, the selection of feature points may be different due to the different imaging methods. For example, the edge in the bright field image may not completely correspond to the position of the fluorescent marker in the fluorescent field image, which makes feature point matching more difficult.

[0077] In this regard, the present application proposes a cross-anchor point alignment method for aligning bright light field and fluorescent field images.

[0078] In a feasible implementation, in S2, image calibration is performed on the bright field image and the fluorescent field image of the cell, including:

[0079] Acquire a bright light field image and a fluorescent field image of the cell, and pre-process the bright light field image and the fluorescent field image;

[0080] A cross-anchor point alignment method is constructed, and image calibration is performed on the two preprocessed images based on the cross-anchor point alignment method to obtain a calibrated image.

[0081] In a feasible implementation manner, preprocessing the bright field image and the fluorescent field image includes:

[0082] Acquire bright field images and fluorescent field images of cells;

[0083] Convert the bright field image of the cell and the fluorescent field image of the cell into a grayscale image;

[0084] Dynamic threshold image enhancement is performed using the adaptive threshold method according to the characteristics of the image, and the image pixels are inverted.

[0085] In a feasible implementation, a cross anchor point alignment method is constructed, and image calibration is performed on the two preprocessed images based on the cross anchor point alignment method to obtain a calibrated image, including:

[0086] Obtain the cell's bright light field preprocessed image and the cell's fluorescence field preprocessed image, use the probabilistic Hough transform algorithm, randomly select edge point pairs for line segment estimation, and extract the funnel-shaped area in the two images;

[0087] Detect the two straight lines on both sides of the funnel shape, extend the two straight lines, and find the intersection of the two extended lines;

[0088] According to the intersection point, the bright light field image and the fluorescent field image are calibrated to obtain a bright light field calibration image and a fluorescent field calibration image, and to obtain an offset between the bright light field image and the fluorescent field image.

[0089] In a feasible implementation, the bright field image and the fluorescent field image are analyzed and it is found that there is a funnel-shaped area in both types of images, such as Figure 2 As shown, the bright field image and the fluorescent field image are aligned according to this area. By detecting the two straight lines on both sides of the funnel, and then extending the straight lines, finding the intersection point as the feature point corresponding to the bright field image and the fluorescent field image, the bright field image and the fluorescent field image are aligned according to this point. The specific steps are as follows: First, the bright field image and the fluorescent field image are preprocessed: converted into grayscale images, and then the dynamic threshold image enhancement is performed using the adaptive threshold method according to the characteristics of the image to improve the image contrast, such as Figure 3 As shown, the left side is the bright field image, and the right side is the fluorescent field image; then the image pixels are inverted to improve the performance of detecting straight lines, as shown in Figure 4 As shown in the figure, the left side is the bright light field image, and the right side is the fluorescent field image. Then, the probabilistic Hough transform algorithm is used to randomly select edge point pairs to estimate the possible line segments, such as Figure 5As shown, the left side is the bright field image, and the right side is the fluorescent field image. Because repeated line segments appear, only the two line segments with the largest slope and the smallest slope are retained, and the line segments are extended to find the intersection point, such as Figure 6 As shown in the figure, the left side is the bright light field image, and the right side is the fluorescent field image. According to the intersection point, the bright light field image and the fluorescent field image are calibrated, as shown in Figure 7 As shown, the left side is the bright light field image, and the right side is the fluorescent field image.

[0090] In a feasible implementation, since there is only one funnel-shaped area in both the bright light field image and the fluorescent field image, the intersection of the straight lines in the bright light field image B1 and the fluorescent field image F1 is the corresponding feature point. According to the relative displacement between the intersection points, the bright light field image and the fluorescent field image can be calibrated to obtain the offset of the two images, which is recorded as offset.

[0091] In a feasible implementation, there are two main methods for image alignment in the prior art, namely alignment based on feature points and alignment based on deep learning methods. Among them, the alignment method based on feature points mainly consists of two methods, SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Up Robust Features), which achieve alignment by detecting and describing local features in the image. The method based on deep learning generally uses a pre-trained CNN model to extract high-level features of the image, and then uses these features for matching and alignment. Although SIFT and SURF are widely used in computer vision, SIFT has high computational complexity. For example, steps such as the need to construct a scale space pyramid, detect extreme points, and assign directional histograms require a large amount of computing resources, which will result in a slow processing speed. In addition, SIFT is very sensitive to changes in lighting, and its performance is poor in scenes that are too bright or too dark (i.e., it is not suitable for the scenes of the present invention). Although SURF is faster than SIFT, the accuracy and discrimination of its feature descriptors are usually not as good as SIFT, and it is also not as robust to changes in illumination intensity or even illumination changes, that is, its performance will drop significantly in scenes that are too bright or too dark. The image alignment method based on deep learning requires a large amount of labeled high-quality data for training, which is costly. The cross-anchor alignment method proposed in the present invention is not only simple and fast, but also can accurately align bright field and fluorescent field images, is less affected by illumination changes, and does not require a large amount of labeled data for training.

[0092] In a feasible implementation manner, in S2, the fluorescence field image detection includes:

[0093] The fluorescence field image is converted into a grayscale image, and is screened according to a preset threshold to remove noise with a grayscale value less than the threshold;

[0094] Convert the filtered image into a binary image;

[0095] A connected domain focusing method was constructed to calculate the 8-connected domain of the target as the coarse detection result of the fluorescent field cells;

[0096] Small connected domains with close center points are merged into large connected domains to obtain the cell coordinates of the fluorescence field.

[0097] In a feasible implementation, the fluorescence field image F2 is converted into a grayscale image, and then a binary image is generated according to a threshold. When converting the binary image, the foreground target pixels are marked as 1 and the background pixels are marked as 0. Then, the 8-connected domain of the target is calculated as the rough detection result of the fluorescent cell. After that, the small connected domains with close center points are merged into a large connected domain to reduce the influence of noise. The detection result is as follows: Figure 8 As shown, the left column is the original image and the right column is the detection result.

[0098] In a feasible implementation, the method currently used for cell detection is mainly based on the detection algorithm improved by the yolo series algorithm, but this type of algorithm requires a large amount of high-quality labeled data for training. In addition, because the cells in the scene of the present invention are small, it is difficult to detect them using the cell detection algorithm based on the yolo series, that is, it is easy to miss detection, resulting in failure to match the bright field cells. The connected domain focusing cell detection method designed by the present invention can detect small-area cells in the fluorescence field image, and can filter out small spots caused by noise, and can accurately detect fluorescent cells in the fluorescence field cells.

[0099] In a feasible implementation, for bright light field cell detection, a detection model (YOLO) is directly trained, and then the detection model is used to directly perform detection to obtain bright light field cell coordinates.

[0100] S3, performing image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image;

[0101] In a feasible implementation, after all three lines have finished running, the bright field cells and the fluorescent field cells are aligned according to the offset, specifically including:

[0102] Obtaining the offset between the bright field image and the fluorescence field image, as well as the fluorescence field cell coordinates and the bright field cell coordinates;

[0103] According to the offset of the bright light field image and the fluorescent field image, as well as the fluorescent field cell coordinates and the bright light field cell coordinates, the detection result images of the bright light field image and the fluorescent field image are aligned to obtain an aligned image.

[0104] S4. Perform fluorescence intensity estimation and color rendering on the aligned image to complete dual-light path image detection and rendering.

[0105] In a feasible implementation, in S4, fluorescence intensity estimation and color rendering are performed on the aligned image to complete dual light path image detection and rendering, including:

[0106] The grayscale value is estimated based on the detection area of ​​the fluorescent field cell and mapped to the fluorescence intensity value range according to the following formula (1):

[0107] (1)

[0108] in, is the fluorescence intensity calculation coefficient, and G is the gray value of the fluorescent cell area.

[0109] According to the fluorescence intensity, green Gaussian spots of different sizes are generated and rendered onto the bright field cells as the visualization results of the fluorescence intensity of the bright field cells. The specific results are as follows: Fig. 9 As shown, dual-light path image detection and rendering of cells are completed.

[0110] In an embodiment of the present invention, a cell image alignment method is provided that can solve the problem that bright light field images and fluorescent field images from different light sources are difficult to align. A cross anchor point alignment method is proposed. By detecting the funnel segments in the two images and extending the segments to find the intersection, the algorithm for aligning the bright light field image and the fluorescent field image can align the bright light field image and the fluorescent field image in real time without being affected by different light sources.

[0111] In the embodiment of the present invention, the provided cell image detection method can address the problem of cell detection in bright light field images and fluorescent field images, and proposes a connected domain focusing method. Gray value filtering is performed according to the cell characteristics of the bright light field image, and the image is converted into a binary image. Then, multiple 8-connected domains are generated through the binary image to focus on the fluorescent cells, and the fluorescent field cells are detected in real time as the detection results of the fluorescent cells.

[0112] In the embodiment of the present invention, the fluorescence intensity of the cells can be calculated according to the image, and rendered into the bright field cells according to the fluorescence intensity, which can provide more accurate data support for cell biology research.

[0113] Fig.10 is a block diagram of a dual-light path image detection and rendering device 300 according to an exemplary embodiment. The device 300 is used for a dual-light path image detection and rendering method. Fig.10 The device includes an image calibration module 310, a fluorescent cell detection module 320, an image alignment module 330 and a fluorescence intensity rendering module 340. Wherein:

[0114] An image calibration module 310 is used to obtain a bright field image of the cell and a fluorescent field image of the cell, and pre-process the bright field image and the fluorescent field image; construct a cross anchor point alignment method, and perform image calibration on the two pre-processed images based on the cross anchor point alignment method to obtain a calibrated image;

[0115] The fluorescent cell detection module 320 is used to construct a connected domain focusing method, and perform fluorescence detection on the bright field image and the fluorescent field image simultaneously based on the connected domain focusing method to obtain a detection result graph;

[0116] An image alignment module 330 is used to perform image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image;

[0117] The fluorescence intensity rendering module 340 is used to perform fluorescence intensity estimation and color rendering on the aligned image, and complete dual-light path image detection and rendering.

[0118] Optionally, the image calibration module 310 is used to calibrate the bright field image and the fluorescent field image of the cell, including:

[0119] Acquire a bright light field image and a fluorescent field image of the cell, and pre-process the bright light field image and the fluorescent field image;

[0120] A cross-anchor point alignment method is constructed, and image calibration is performed on the two preprocessed images based on the cross-anchor point alignment method to obtain a calibrated image.

[0121] Optionally, preprocessing the bright field image and the fluorescent field image includes:

[0122] Acquire bright field images and fluorescent field images of cells;

[0123] Convert the bright field image of the cell and the fluorescent field image of the cell into a grayscale image;

[0124] Dynamic threshold image enhancement is performed using the adaptive threshold method according to the characteristics of the image, and the image pixels are inverted.

[0125] Optionally, constructing a cross anchor point alignment method, performing image calibration on the two preprocessed images based on the cross anchor point alignment method to obtain a calibrated image, including:

[0126] Obtain the cell's bright light field preprocessed image and the cell's fluorescence field preprocessed image, use the probabilistic Hough transform algorithm, randomly select edge point pairs for line segment estimation, and extract the funnel-shaped area in the two images;

[0127] Detect the two straight lines on both sides of the funnel shape, extend the two straight lines, and find the intersection of the two extended lines;

[0128] According to the intersection point, the bright light field image and the fluorescent field image are calibrated to obtain a bright light field calibration image and a fluorescent field calibration image, and to obtain an offset between the bright light field image and the fluorescent field image.

[0129] Optionally, the fluorescent cell detection module 320 is used to convert the fluorescent field image into a grayscale image, filter it according to a preset threshold, and remove the noise with a grayscale value less than the threshold;

[0130] Convert the filtered image into a binary image;

[0131] A connected domain focusing method was constructed to calculate the 8-connected domain of the target as the coarse detection result of the fluorescent field cells;

[0132] Small connected domains with close center points are merged into large connected domains to obtain the cell coordinates of the fluorescence field.

[0133] Optionally, the fluorescent cell detection module 320 is used to perform YOLO detection on the bright light field image to obtain bright light field cell coordinates.

[0134] Optionally, the image alignment module 330 is used to obtain the offset of the bright field image and the fluorescent field image, as well as the fluorescent field cell coordinates and the bright field cell coordinates;

[0135] According to the offset of the bright light field image and the fluorescent field image, as well as the fluorescent field cell coordinates and the bright light field cell coordinates, the detection result images of the bright light field image and the fluorescent field image are aligned to obtain an aligned image.

[0136] Optionally, the fluorescence intensity rendering module 340 is used to estimate the grayscale value according to the detection area of ​​the fluorescence field cell and map it to the fluorescence intensity value range according to the following formula (1):

[0137] (1)

[0138] in, is the fluorescence intensity calculation coefficient, and G is the gray value of the fluorescent cell area.

[0139] In the embodiment of the present invention, the image calibration module 310 and the image alignment module 330 can solve the problem that it is difficult to align bright light field images and fluorescent field images from different light sources, and propose a cross anchor point alignment method. By detecting the funnel segments in the two images and extending the segments to find the intersection, the algorithm is used to align the bright light field image and the fluorescent field image. The bright light field image and the fluorescent field image can be aligned in real time without being affected by different light sources.

[0140] In the embodiment of the present invention, the fluorescent cell detection module 320 can propose a connected domain focusing method for the bright light field image and fluorescent field image cell detection problem, perform gray value filtering according to the bright light field image cell characteristics, convert it into a binary image, and then generate multiple 8-connected domains through the binary image to focus on the fluorescent cells, and use them as the detection results of the fluorescent cells to detect the fluorescent field cells in real time.

[0141] In the embodiment of the present invention, the fluorescence intensity rendering module 340 can calculate the fluorescence intensity of the cells according to the image, and render the cells in the bright field according to the fluorescence intensity, thereby providing more accurate data support for cell biology research.

[0142] Fig.11 is a schematic diagram of the structure of a dual-light path image detection and rendering device provided by an embodiment of the present invention, such as Fig.11 As shown, the dual-light path image detection and rendering device may include the above Fig.10 Optionally, the dual-light path image detection and rendering device 410 may include a first processor 2001 .

[0143] Optionally, the dual-light-path image detection and rendering device 410 may further include a memory 2002 and a transceiver 2003 .

[0144] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0145] Combine the following Fig.11 The components of the dual-light path image detection and rendering device 410 are specifically introduced:

[0146] The first processor 2001 is the control center of the dual-light path image detection and rendering device 410, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0147] Optionally, the first processor 2001 may execute various functions of the dual-light path image detection and rendering device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0148] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig.11 CPU0 and CPU1 are shown in FIG.

[0149] In a specific implementation, as an embodiment, the dual-light path image detection and rendering device 410 may also include multiple processors, such as Fig.11 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0150] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0151] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and access the first processor 2001 through the interface circuit ( Fig.11 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0152] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0153] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig.11 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0154] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Fig.11 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0155] It should be noted that Fig.11 The structure of the dual-light path image detection and rendering device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0156] In addition, the technical effects of the dual-light path image detection and rendering device 410 can refer to the technical effects of the dual-light path image detection and rendering method described in the above method embodiment, which will not be repeated here.

[0157] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0158] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0159] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0160] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0161] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0162] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0165] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0166] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-light path image detection and rendering method, characterized in that: The method comprises: S1. Build a multi-threaded processing flow; obtain bright field images and fluorescent field images of cells; S2. Based on the multi-threaded processing flow, image calibration, bright light field image detection and fluorescence field image detection are performed on the bright light field image and the fluorescence field image of the cell at the same time to obtain a calibration image, a bright light field image detection result and a fluorescence field image detection result; S3, performing image alignment based on the detection result images of the bright light field image and the fluorescent field image to obtain an aligned image; S4. Perform fluorescence intensity estimation and color rendering on the aligned image to complete dual-light path image detection and rendering.

2. The dual-light path image detection and rendering method according to claim 1, characterized in that: In S2, image calibration is performed on the bright field image and the fluorescent field image of the cell, including: Acquire a bright light field image of the cell and a fluorescent field image of the cell, and preprocess the bright light field image and the fluorescent field image; A cross-anchor point alignment method is constructed, and image calibration is performed on the two preprocessed images based on the cross-anchor point alignment method to obtain a calibrated image.

3. The dual-light path image detection and rendering method according to claim 2, characterized in that: Preprocessing the bright light field image and the fluorescent field image includes: Acquire bright field images and fluorescent field images of cells; Convert the bright field image of the cell and the fluorescent field image of the cell into a grayscale image; Dynamic threshold image enhancement is performed using the adaptive threshold method according to the characteristics of the image, and the image pixels are inverted.

4. The dual-light path image detection and rendering method according to claim 1, characterized in that: The constructing of the cross anchor point alignment method, performing image calibration on the two preprocessed images based on the cross anchor point alignment method, and obtaining a calibrated image, comprises: Obtain the cell's bright light field preprocessed image and the cell's fluorescence field preprocessed image, use the probabilistic Hough transform algorithm, randomly select edge point pairs for line segment estimation, and extract the funnel-shaped area in the two images; Detect the two straight lines on both sides of the funnel shape, extend the two straight lines, and find the intersection of the two extended lines; According to the intersection point, the bright light field image and the fluorescent field image are calibrated to obtain a bright light field calibration image and a fluorescent field calibration image, and to obtain an offset between the bright light field image and the fluorescent field image.

5. The dual-light path image detection and rendering method according to claim 4, characterized in that: In S2, the fluorescence field image detection includes: The fluorescence field image is converted into a grayscale image, and is screened according to a preset threshold to remove noise with a grayscale value less than the threshold; Convert the filtered image into a binary image; A connected domain focusing method was constructed to calculate the 8-connected domain of the target as the coarse detection result of the fluorescent field cells; Small connected domains with close center points are merged into large connected domains to obtain the cell coordinates of the fluorescence field.

6. The dual-light path image detection and rendering method according to claim 5, characterized in that: In S2, bright field image detection includes: The bright light field image is subjected to YOLO detection to obtain the bright light field cell coordinates.

7. The dual-light path image detection and rendering method according to claim 6, characterized in that: In S3, performing image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image includes: Obtaining the offset between the bright field image and the fluorescence field image, as well as the fluorescence field cell coordinates and the bright field cell coordinates; According to the offset of the bright light field image and the fluorescent field image, as well as the fluorescent field cell coordinates and the bright light field cell coordinates, the detection result images of the bright light field image and the fluorescent field image are aligned to obtain an aligned image.

8. The dual-light path image detection and rendering method according to claim 7, characterized in that: In S4, the fluorescence intensity of the aligned image is estimated and color rendered to complete dual light path image detection and rendering, including: The grayscale value is estimated based on the detection area of ​​the fluorescent field cell and mapped to the fluorescence intensity value range according to the following formula (1): （1） in, is the fluorescence intensity calculation coefficient, G is the gray value of the fluorescent cell area; According to the fluorescence intensity, Gaussian spots of different sizes are generated and rendered onto the bright light field cells as the fluorescence intensity visualization results of the bright light field cells, completing the dual-light path image detection and rendering of the cells.

9. A dual-light path image detection and rendering device, the dual-light path image detection and rendering device is used to implement the dual-light path image detection and rendering method according to any one of claims 1 to 8, characterized in that: The device comprises: An image calibration module is used to obtain a bright field image of a cell and a fluorescent field image of a cell, and preprocess the bright field image and the fluorescent field image; construct a cross anchor point alignment method, and perform image calibration on the two preprocessed images based on the cross anchor point alignment method to obtain a calibrated image; A fluorescent cell detection module, used for constructing a connected domain focusing method, and performing fluorescence detection on the bright light field image and the fluorescent field image simultaneously based on the connected domain focusing method to obtain a detection result graph; An image alignment module, used to perform image alignment based on the detection result images of the bright field image and the fluorescent field image to obtain an aligned image; The fluorescence intensity rendering module is used to perform fluorescence intensity estimation and color rendering on the aligned image to complete dual-light path image detection and rendering.

10. A dual-light path image detection and rendering device, characterized in that: The robot system control device of the model-free regression reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images